JP2024082441A - Image processing device, printing system, and image processing program - Google Patents

Abstract

To accurately determine and display how an image printed on a print medium will look on the print medium.
[Solution] An image processing device specifies printing conditions including the type of print medium, and inputs print image data including pixel values of each pixel of a print image to be formed on the print medium. Then, texture parameters representing the texture of the print medium for which the printing conditions have been specified are set in consideration of the pixel values of the print image data at the location where the print image is to be formed. A rendering execution unit performs rendering processing using the set texture parameters and the print image data, and generates a rendered image of the print medium on which the print image has been formed.
[Selected Figure] Figure 1

Description

This disclosure relates to image processing techniques that can display how the print medium on which an image is printed will look.

Image processing techniques that express the fact that the appearance of an image changes depending on differences in lighting on an object have been known for some time. For example, the following Patent Document 1 discloses a technique that corrects the glossiness of an image according to the attributes of the illuminated partial area (face, mouth, teeth, etc.) when applying a virtual lighting effect to the image.

JP 2016-213718 A

However, Patent Document 1 only corrects the appearance of partial areas in an image, particularly the glossiness, and does not consider at all how the print medium on which such an image is printed appears.

The present disclosure can be realized in the following manner.
(1) One embodiment of the present disclosure is an image processing device. The image processing device includes a printing condition specification unit that specifies printing conditions including a type of printing medium, an image input unit that inputs printing image data including pixel values of each pixel of a printing image formed on the printing medium, a texture setting unit that sets texture parameters representing the texture of the printing medium for which the printing conditions are specified, taking into account the pixel values of the printing image data at a location where the printing image is formed, and a rendering execution unit that generates a rendering image of the printing medium on which the printing image is formed by rendering processing using the printing image data and the texture parameters.

(2) Another embodiment of the present disclosure is an image processing program that generates a rendering image of a print medium on which an image is printed. This image processing program realizes, by a computer, a first function of identifying printing conditions including the type of the print medium, a second function of inputting print image data including pixel values of each pixel of a print image to be formed on the print medium, a third function of setting texture parameters representing the texture of the print medium for which the printing conditions have been identified, taking into account the pixel values of the print image data at the location where the print image is formed, and a fourth function of generating a rendering image of the print medium on which the print image is formed by rendering processing using the print image data and the texture parameters.

1 is a schematic configuration diagram showing an image processing apparatus according to an embodiment. 11 is a flowchart showing an outline of an image display process. 11 is a flowchart showing an example of a color conversion process. FIG. 4 is an explanatory diagram showing the logical configuration of a rendering execution unit together with a material texture setting unit according to the first embodiment. 11 is a flowchart showing an overview of a texture setting process. FIG. 4 is an explanatory diagram showing the setting of an image of a print medium and texture parameters. FIG. 1 is an explanatory diagram illustrating a display example of a print medium on which an image is printed. 1 is an explanatory diagram showing the relationship between a light source, a viewpoint, and the angle of a surface of a 3D object. 1 is an explanatory diagram showing a schematic diagram of how the display on a print medium changes depending on a change in the angle of the surface of the print medium on which an image is formed relative to a light source. 11A and 11B are explanatory diagrams showing a comparison between a case where pixel values are not taken into consideration as texture and a case where they are not taken into consideration; 10 is a flowchart showing a modified example of the texture setting process of the first embodiment. 13A and 13B are explanatory diagrams showing a comparison between a case in which pixel values are not taken into consideration as texture and a case in which texture is not taken into consideration in a modified example. FIG. 11 is an explanatory diagram showing the logical configuration of a rendering execution unit together with a material texture setting unit according to a second embodiment. FIG. 13 is an explanatory diagram showing the logical configuration of a rendering execution unit together with a material texture setting unit according to a third embodiment. FIG. 4 is a diagram illustrating an example of a texture table. FIG. 13 is a schematic configuration diagram of a printing system according to a fourth embodiment. 11A to 11C are explanatory diagrams showing examples of other print media on which the print result appears.

A. First embodiment:
(A1) Hardware configuration:
FIG. 1 shows a schematic configuration of an image processing device 100 according to the present embodiment. The image processing device 100 performs image processing to preview the appearance of an image printed on a predetermined print medium. The image processing device 100 not only performs image processing, but also displays the processing result as a preview image. As shown in the figure, the image processing device 100 includes an image data input unit 90 for inputting print image data, a color management system 111 for color conversion, a rendering execution unit 121 for rendering the print medium, a texture setting unit 131, a printing condition specification unit 135, an image memory 139, and an image display unit 151 for displaying a preview image. Programs for performing each process described below are stored in a storage unit 171 or the like of the image processing device 100, and each function of the image processing device 100 is realized by a CPU or GPU (not shown) executing the programs stored in the storage unit 171.

For simplicity, the color management system may be abbreviated as CMS below. The CMS 111 is capable of acquiring print image data ORG representing the image to be printed (hereinafter referred to as the print image) via the image data input unit 90. When inputting the image data ORG, the image data input unit 90 may receive the image data ORG via wired or wireless communication from the image forming device that created the image data ORG, or may read the image data ORG from a memory card that stores the print image data ORG in file format. Of course, the image data may be acquired via a network. Alternatively, the image data ORG may be created within this image processing device 100.

The CMS 111 converts the color of the print image to be previewed into the object color to be displayed on the print medium. The converted image data is called managed image data MGP. Details of the CMS process will be described later. The managed image data MGP is set as the texture of the print medium, which is a 3D object. The input profile IP, media profile MP, common color space profile CP, and the like are input to the CMS 111. The input profile IP is used to convert from a device-dependent input color system, such as RGB data, to a device-independent color system such as L ^* a ^* b ^* (hereinafter simply abbreviated as Lab).

The media profile MP is a profile that represents the color reproducibility when printing is performed on a specific printing medium by a specific printing device such as a printer under printing conditions such as a specific printing resolution, and is a profile that converts color values between a device-independent color system and a device-dependent color system. The media profile MP also includes information such as the print settings of the printing device other than the print medium. Therefore, if it is attempted to cover all combinations of the printing device (printer) x print medium x print settings, the number of types of media profile MP will increase. Therefore, when the dependency of the printing conditions is small or when it is not desired to increase the number of profiles, the media profile MP is configured as a combination of the printing device (printer) x print medium. In this way, since the color of the image on the print medium (media) is related to the characteristics of the printing device and the characteristics of the print medium itself, the media profile MP may be referred to as the print profile MP below. In this embodiment, an inkjet printer is assumed as the printer that prints the image data ORG. In the case of an inkjet printer, the media profile may also differ depending on the type of ink used, such as the ink set.

In this embodiment, the designation of such a combination of printing device and printing medium is called printing condition PCC. In the first embodiment, the printing condition specification unit 135 receives printing condition PCC input by the user and selects a media profile MP that the CMS 111 uses to convert the image data. In other embodiments described below, such printing condition PCC may be referenced by the texture setting unit 131. For this reason, in FIG. 1, the relationship in which the printing condition specification unit 135 gives instructions to the texture setting unit 131 based on the printing condition PCC is shown by a dashed line. The process in which the texture setting unit 131 sets texture based on the printing condition PCC will be described in detail in the third embodiment.

When the input profile IP is applied to the image data ORG, and then the media profile MP is applied, the color values obtained when printed under specific printing conditions, that is, the color values are dependent on the printing device and printing medium. When the media profile MP is applied to the color values of this image to convert them from a device-dependent color system to a device-independent color system, and then the common color space profile CP is applied, they are converted to an expression in a second color space (here, the sRGB color space) used during rendering. Since the media profile MP is used to convert the color values to values that are dependent on the characteristics of the printing device and printing medium, the image data ORG is color-converted to a range of color values that can actually be printed. The common color space profile CP is used to convert the image data to color values in the color space used during rendering. The sRGB color space is a typical common color space, but AdobeRGB, Display-P3, etc. may also be used.

As described above, the CMS 111 uses each profile to convert image data ORG expressed in the first color space, which is a device-dependent color system, into image data (managed image data) MGP expressed in the sRGB color space, which is the second color space used during rendering. Here, the converted image data is not limited to color values in the sRGB color space, and may be expressed in any color space as long as it is expressed in a color space that the rendering execution unit 121 can handle. For example, if the rendering execution unit 121 is configured to be able to render using color values and spectral reflectance in the Lab or XYZ color space, the image data may be converted into color values used when displayed on the image display unit 151 in the lighting process (described later) performed in the rendering execution unit 121 or in a post-processing unit (described later) downstream of the rendering execution unit 121.

The rendering condition data RCD, which is the rendering condition in the rendering execution unit 121, is set by the rendering condition setting unit 125 and referenced by the rendering execution unit 121. The rendering condition data RCD includes 3D object information of the print medium, camera information such as the position from which the print medium is viewed, lighting information such as the lighting position and color, and background information indicating information about the background on which the print medium is placed. The details and handling of this rendering condition data RCD will be explained in detail later.

The texture setting unit 131 sets the texture of the image to be displayed on the print medium. The textures handled by the texture setting unit 131 will be explained in detail later, but they include the smoothness S and metallicity M of the surface of the print medium. Even when the same image is printed, the appearance will change depending on the smoothness and metallicity of the print medium surface.

These printing conditions PCC and rendering condition data RCD, representative data with a certain frequency of use or more, may be stored in advance in a non-volatile manner in the printing condition specification unit 135 and the rendering condition setting unit 125, selected as necessary, and referred to by the CMS 111 and the rendering execution unit 121. Texture data for printing media that are not usually used, such as those that are used infrequently, such as cloth, cans, and plastic sheets, may be stored on an external site and acquired when necessary. The rendering condition data RCD, such as lighting information, may be individually specified by the user at the time of rendering, but representative camera angles and light sources may be stored in the rendering condition setting unit 125 and used. The camera angle refers to the position and direction from which the target printing medium is viewed, and corresponds to the position of the virtual viewpoint and the direction of the line of sight of the user viewing the virtual space. For this reason, the camera may be described as the viewpoint or direction of the line of sight, and may be referred to as the "viewpoint" or "view".

The image display unit 151 displays the image of the print medium rendered by the rendering execution unit 121 together with the background, etc. The image display unit 151 reads and displays the image data for display that the rendering execution unit 121 has written to the image memory 139. In this embodiment, the image display unit 151 is provided in the image processing device 100, but the image processing device 100 may be configured not to include the image display unit 151. In that case, when a user wishes to see a preview image, the rendered data is obtained and transferred to another device capable of displaying it for reference. For example, an independent tablet wirelessly connected to the image processing device 100 may be used as the display unit, and the rendering results may be previewed on the tablet.

The image processing device 100 may be realized as a dedicated machine, or may be realized by executing an application program on a computer. Of course, computers also include terminals such as tablets and mobile phones. Since the processing of the rendering execution unit 121 requires considerable resources and computing power, the image processing device 100 may be configured such that only the rendering execution unit 121 is executed by a CPU capable of high-speed processing or a dedicated GPU, and the rendering execution unit 121 is located at another site on the network.

(A2) Overview of image processing:
The image display process executed by the image processing device 100 will be described with reference to FIG. 2A. When the image processing device 100 is instructed to display on the image display unit 151 how the print medium on which the image is printed looks, the image processing device 100 starts the process shown in the figure and first performs a process of acquiring image data (step S110). The image processing device 100 acquires image data ORG, which is data of the image to be printed on the print medium when displaying the printed state, via the data input unit 90. Next, the printing conditions PCC are input and a process of specifying the printing conditions is performed (step S120). In this embodiment, the printing conditions PCC are the printing device to be used for printing and the type of printing medium to be used for printing. As described above, if there are multiple ink sets available in the printing device, the ink sets (ink types) are also input as printing conditions PCC, and the printing conditions are specified based on this.

Next, various profiles are set (step S130). Specifically, an input profile IP is set according to the format of the acquired image data ORG, that is, the color space used to express the image data ORG, or a media profile MP is set according to the printing conditions set based on the printing conditions PCC such as the type of printing device and printing medium. Then, a color conversion process is performed by CMS111 (step S140). An example of the color conversion process is shown in FIG. 2B. The figure is a flowchart showing the process of converting the print image data ORG into color data of a common color space for rendering processing by CMS111. When the color conversion process is started, first, the print image data ORG and the input profile IP are input, and the print image data ORG (hereinafter abbreviated as image data ORG) expressed in a device-dependent color system (e.g., RGB color system) is converted into color data of a device-independent color system (e.g., Lab or XYZ color system) (step S141). Next, it is determined whether a media profile MP is prepared (step S142), and if a media profile MP is prepared, it is applied, and color conversion to the range of colors that can be expressed by printing is performed, taking into account the combination of printing device (printer) and printing medium as the printing conditions (step S143). If a media profile MP is not prepared, the processing of step S143 is not performed. After that, using a common color space profile CP, conversion is performed to color values of a common color space, which is a second color space used during rendering (step S144). In this embodiment, sRGB is used as the common color space. The managed image data MGP obtained in this way is set as the albedo color, which is the texture of the 3D object (step S145), and the color conversion processing ends.

In step S143, if the rendering intent for color conversion of the media profile is set to absolute, the color of the print medium itself (background color) can be reflected. Note that if the color values of the image to be color converted in step S145 are outside the gamut of the sRGB color space, they may be approximated to values within the sRGB color space, or may be treated as if they could take values outside the sRGB gamut. The RGB values of image data are generally stored as 8 bits for each color, that is, integers with values from 0 to 255. However, if pixel values are instead represented as floating-point numbers with values from 0.0 to 1.0, values outside the sRGB gamut can be treated as negative values or values exceeding 1.0.

The color conversion by CMS111 is not limited to the configuration shown in FIG. 2B, and can be performed by other configurations. For example, if correction data DPD is prepared for correcting the shift in the display color of image display unit 151 relative to the common color space sRGB, a configuration may be used in which, after color conversion by media profile MP in FIG. 2B (step S143), color conversion processing is performed using display device correction data prepared for correcting the characteristics of the display device.

It is also possible to prepare composite correction data by combining such display device correction data with the common color space profile CP in advance, and perform color conversion using the composite correction data instead of color conversion using the common color space profile CP (step S145). Note that correction of the display color shift of the image display unit 151 may be performed in the post-processing unit PST after the render backend shown in FIG. 3 described later, instead of in the CMS 111.

After the color conversion process (step S140), the texture parameter TXT is input (step S150), and a texture setting process (step S160) is performed to set the texture according to the texture parameter TXT and the printing conditions already specified. In this embodiment, the texture setting process is a process for setting the texture of the image printed on the print medium to an appropriate texture, which is influenced not only by texture parameters such as the smoothness and metallicity of the surface of the print medium, but also by the pixel values of the image to be printed. Details will be described later.

After the texture has been set, physically based rendering is performed using the managed image data MGP that has been color converted using the CMS 111 and the texture data that has been set (step S170), and the rendering results are displayed on the image display unit 151 (step S180). This processing routine then ends.

(A3) Configuration of rendering execution unit and rendering process:
The rendering process of step S170 described above will be described. The rendering process is executed by the rendering execution unit 121. The rendering execution unit 121 renders the managed image data MGP that the CMS 111 converts color and outputs, and displays on the image display unit 151 how the print medium on which the image data ORG is printed looks in the virtual space. An example of the configuration of the rendering execution unit 121 is shown in FIG. 3. This rendering execution unit 121 shows a representative configuration for performing a physically based rendering process, and other configurations can also be adopted. The rendering execution unit 121 of this embodiment adopts a pipeline configuration including a vertex pipeline VPL and a pixel pipeline PPL, and executes physically based rendering at high speed. The vertex pipeline VPL includes a vertex shader VS and a geometry shader GS. It should be noted that a configuration without using the geometry shader GS is also possible.

The vertex shader VS converts the coordinates on the print medium of the vertices of the print medium, which is a 3D object, into coordinates in the three-dimensional space to be rendered. Coordinate conversion comprehensively includes coordinate conversion such as coordinates of the model to be rendered (here, the print medium) → world coordinates → view (camera) coordinates → clip coordinates, but conversion to view coordinates is performed by the geometry shader GS. The vertex shader VS also performs shading and calculation of texture coordinates (UV). When performing these processes, the vertex shader VS and geometry shader GS refer to 3D object information TOI, camera information CMR, lighting information LGT, and even background information BGD as rendering condition data RCD prepared in advance.

The 3D object information TOI is information about the shape of the print medium as a 3D object. Since a real print medium is not a flat surface, it is basically treated as a collection of tiny polygons, but if the surface of the print medium is expressed with tiny polygons, the number of polygons becomes enormous. For this reason, it is also realistic to handle the surface of the print medium using textures such as normal maps and height maps. Textures such as normal maps and height maps are given as texture parameters, which will be described later. The camera information CMR is virtual information about the position and direction in which the camera is installed relative to the print medium. The lighting information LGT includes at least one of the virtual information such as the position, angle, strength, and color temperature of the light source in the virtual space in which the print medium is placed. Note that it is also possible to set multiple light sources. In this case, the effects of the multiple light sources can be calculated separately and superimposed on the 3D object.

Although the background information BGD is not necessary, it is information about the background in which the print medium is placed as a 3D object in the virtual space. The background information BGD includes information about objects such as walls, floors, and furniture placed in the virtual space, and these objects are the subject of rendering in the rendering execution unit 121 in the same way as the print medium. In addition, since lighting hits these background objects and illuminates the print medium, they are also treated as part of the lighting information. Rendering using these various types of information makes it possible to provide a three-dimensional preview. The vertex information calculated by the vertex shader VS is passed to the geometry shader GS.

The geometry shader GS is used to process a set of vertices within an object. The geometry shader GS makes it possible to increase or decrease the number of vertices at run time, and to change the type of primitives that make up a 3D object. One example of increasing or decreasing the number of vertices is the culling process. In the culling process, vertices that are not visible to the camera are excluded from the processing target based on the camera's position and direction. The geometry shader GS also performs processing such as generating new primitives from existing primitives such as points, lines, and triangles. The geometry shader GS inputs a primitive that contains information on the entire primitive or adjacent primitives from the vertex shader VS. The geometry shader GS processes the input primitive and outputs the primitive to be rasterized.

The output of the vertex pipeline VPL, specifically the primitives processed by the geometry shader GS, is rasterized by the rasterizer RRZ, converted into pixel-by-pixel data, and passed to the pixel pipeline PPL. In this embodiment, the pixel pipeline PPL includes a pixel shader PS and a render backend RBE.

The pixel shader PS operates on rasterized pixels, and in short, calculates the color for each pixel. Based on the information input from the vertex shader VS and the geometry shader GS, it performs texture synthesis processing and surface color application processing. The pixel shader PS maps the managed image data MGP, which is the image data ORG converted by the CMS 111 based on various profiles, onto the print medium as a 3D object. At this time, the lighting processing function provided in the pixel shader PS performs lighting processing based on the light reflection model of the object and the above-mentioned lighting information LGT, and performs mapping of the managed image data MGP. Since the entire print medium as a 3D object is handled, the managed image data MGP to be mapped is an image that targets the entire print medium, including the image formed by ink on the print medium. The reflection model used in the lighting processing is one of the calculation formulas of a mathematical model for simulating lighting phenomena in the real world. The reflection model used in this embodiment will be explained in detail later.

(A4) Texture setting process:
In this embodiment, a smoothness modification value S, which is one of the texture parameters TXT, is written to the alpha channel α-ch of the image data MGP to be processed by the pixel shader PS. The smoothness modification value S is set by the texture setting unit 131 based on the texture parameter TXT and the pixel values of the image data MGP, and written to the image data MGP. The function for determining the smoothness modification value S is shown in FIG. 3 as a modification value calculation unit 132. Details of the process (FIG. 2A, step S160) performed by the texture setting unit 131 including the modification value calculation unit 132 will be described with reference to FIG. 4.

In the texture setting process performed by the texture setting unit 131, first, the managed image data MGP is read (step S161). Next, the texture parameter TXT and the printing conditions PCC are read (step S162). The texture parameter TXT includes smoothness and metallicity, but here, the smoothness is focused on, and the initial value S1 set for the printing medium is set as the smoothness based on the type of printing medium included in the printing conditions PCC (step S163). In general, if the printing medium is, for example, a plastic sheet or glossy photo paper, the initial value S of smoothness is set high, and if it is plain paper or fabric, it is set low. In this embodiment, the initial value S1 of smoothness is a value that corresponds to the smoothness of the paper itself, which is relatively smooth, such as glossy printing paper, and is set to, for example, a value of 0.95. Here, smoothness is taken as the texture parameter TXT, but it may be another texture parameter TXT, or multiple parameters may be set. The other texture parameters will be described in detail later as part of the rendering process.

After the above preparations are made, the processes of steps S164 to S168 are repeated for all pixels corresponding to the print medium (STs to STe). All pixels of the managed image data MGP refer to the entire print medium. The reading order can be any order; for example, the upper left corner of the print medium can be used as the origin to read out along the width direction (main scanning direction) of the print medium to the edge, and this can be repeated sequentially along the length direction (sub-scanning direction) of the print medium until it reaches the bottom right edge of the print medium.

In any case, one pixel is focused on, the pixel value of that pixel is read (step S164), and the smoothness correction value S for that pixel is calculated from the read pixel value (step S165). Since the managed image data MGP is sRGB data corresponding to the ink CMYK ejection amount of the printing device by the CMS 111, a pixel with an sRGB value where the sum of the ink ejection amount is zero can be determined to be paper white. Therefore, it is determined whether the pixel read from the managed image data MGP is paper white, that is, whether it is a pixel where ink is not ejected onto the printing medium, or a pixel where ink is ejected (step S165a), and if the result is the former, the smoothness correction value S is set to the initial value S1 set in advance (step S165b), and if the result is the latter, the smoothness correction value S is set to a value obtained by multiplying the initial value S1 set in advance by the gain g (step S165c). In this embodiment, the gain g is set to a value less than 1, such as 0.9, so the smoothness correction value S for pixels onto which ink is ejected is set to approximately 0.85. In contrast, the smoothness of pixels in the white portion of the paper onto which ink is not ejected is set to the initial value S1, 0.95. After calculating the smoothness correction value S for each pixel in this way, this value is written to the alpha channel α-ch of the managed image data MGP (step S168). The above process (steps S164 to S168) is repeated for all pixels of the managed image data MGP (STs to STe).

An example of the process for setting the smoothness correction value S described above is shown in FIG. 5. The upper part of the figure shows an example of managed image data MGP. In this example, the managed image data MGP has an area of horizontal stripes of color values CK, CB, CR, and CY near the center of the printing paper. Each of these areas is considered to be an image in which CMYK ink is ejected at a predetermined density by color conversion of the image data ORG by CMS111. It also shows that the initial value S1 of the smoothness of the printing medium itself is set to a value of 0.95. In this example, the smoothness of the printing medium surface is likely to change depending on the type of ink ejected on the printing medium and the total amount of ink ejected. In contrast, in the above process, the smoothness is changed depending on whether or not it is a part where ink for image formation is ejected, so this process is called smoothness correction, and the set value is called smoothness correction value S. In this specification, correction is treated as a type of correction. Although it would be possible to perform a highly accurate calculation to determine smoothness from pixel values and then correct the smoothness to obtain a correction value, a correction that can show the effect of pixel values in the rendered image is sufficient.

The middle part of the figure shows the gain g value for each area. In the area where ink is ejected, the gain g is set to a value of 0.9, and elsewhere, the gain g is not set. The bottom part of the figure shows the smoothness correction value S calculated for the area corresponding to the pixels that make up the managed image data MGP from the initial smoothness value S1 and gain g set for the printing medium itself. The smoothness correction value S for the area corresponding to the pixels where ink is ejected is calculated as S1 x g, giving a value of about 0.85, and the smoothness correction value S for the paper white area where ink is not ejected is calculated as the original initial smoothness S1, with a 0.95 mark. Note that the gain g for the paper white area may be set to a value of 1.0, and S1 x g may be calculated in the same way as for the area where ink is ejected.

In this embodiment, the smoothness of each pixel of the managed image data MGP is calculated using the initial smoothness value S1 determined for the entire print medium and the gain g prepared according to the part where the ink is ejected. Among these, the initial smoothness value S1 and the gain g may be prepared as part of the print condition PCC, or may be input as part of the texture parameter TXT. If the smoothness of the print medium itself or the decrease in smoothness due to ejecting ink is determined by the type of ink used by the print medium or the printing device, it can be treated as the print condition PCC, and if it is to be set by the user as one of the parameters that determine the appearance, it can be specified as part of the texture parameter TXT. In addition, without using the gain g, the value of the smoothness correction value S may be uniquely set depending on whether the pixel is in the white part of the paper or not. The gain g may be related to the pixel value or the luminance obtained from the pixel value, and the gain g may be obtained for each pixel, and the smoothness correction value S of the part corresponding to that pixel may be calculated. The processing in this case will be described later as a modified example. As described above, the gain g may be input from the outside as part of the printing conditions PCC or texture parameters TXT, or may be held as a fixed value within the program. Alternatively, a table may be prepared that defines the gain g according to the type of printing medium or ink, and this may be read from the outside or held within the program.

In this embodiment, the smoothness correction value S calculated in step S165 of FIG. 4 is written in step S168 as part of the managed image data MGP, that is, in the alpha channel α-ch corresponding to the pixel. Then, when the setting process (STs to STe) for all pixels is completed, the correction value S is written in the alpha channel α-ch of the managed image data MGP and output to the pixel shader PS for each managed image data MGP (step S169). Note that the smoothness correction value S may also be written in a separate file having a data structure associated with the coordinates of the pixels of the managed image data MGP and passed to the pixel shader PS.

(A5) Display of rendering image:
The pixel pipeline PPL of the rendering execution unit 121 receives the managed image data MGP with the modification value S written therein, and performs processing to manipulate the pixels. This processing becomes a heavy load and takes a long time when the number of pixels after rasterization increases, for example, when the output resolution is high. For this reason, the processing takes more time than processing on a vertex basis, and the efficiency of the pipeline processing may become insufficient. In this embodiment, the processing program of the pixel shader PS is optimized for execution on a GPU with high parallel processing performance, thereby realizing advanced effects including texture expression in a short time.

The render backend RBE then determines whether the pixel information obtained by processing the pixel shader PS should be written to the image memory 139 for display. Only when the render backend RBE determines that it is OK to write the pixel data to the image memory 139 is the pixel data saved as something to be drawn. Tests used to determine whether to write include the well-known "alpha test," "depth test," and "stencil test." The render backend RBE executes the set test from among these tests and writes the pixel data to the image memory 139.

The rendering pipeline process is completed with the above processing, and then the post-processing unit PST processes the data stored in the frame memory FM to improve the appearance. One example of such processing is anti-aliasing, which removes unnecessary edges from the image to make it smoother. Other processes include ambient occlusion, screen space reflection, and depth of field, and the post-processing unit PST can be configured to perform the necessary post-processing.

When the rendering execution unit 121 performs the above processing, rendering is completed and the result is output as the render result RRD. In practice, the data written to the image memory 139 is read out in accordance with the display cycle of the image display unit 151, and is displayed as the render result RRD. An example of the render result RRD is shown in FIG. 6. In this example, the image display unit 151 displays the print medium PLb as a 3D object placed in the virtual space, the light source LG, and a background object Bob such as furniture that exists as part of the background.

Figure 7 illustrates the relationship between the print medium PLb placed in a virtual space and the light source LG and the viewpoint (camera) VP. The relationship between the light source LG, the viewpoint VP, and the print medium PLb is three-dimensional in the virtual space VSP, but the figure shows the virtual space VSP on an x-z plane. x is the coordinate of the point where each vector described below is gathered. The positional relationship of the light source LG and the viewpoint VP that illuminates the print medium PLb, which is the target of rendering, is illustrated with respect to a specific coordinate x of the print medium PLb that is the target of rendering. The figure illustrates the light source direction vector ωl from the coordinate x to the light source LG, the viewpoint direction vector ωv from the coordinate x to the viewpoint VP, and the half vector HV of both. In addition, the symbol Np indicates the normal vector when it is assumed that the print medium PLb is a perfect plane PLp, and the symbol Nb indicates the normal vector at the coordinate x of the actual print medium PLb that is not a perfect plane. Note that in Figure 6, the viewpoint VP (camera) is assumed to be almost in front of the print medium PLb, and the rendering result of the print medium PLb is illustrated.

In the image processing device 100 of this embodiment, the position and angle of the print medium in the virtual space can be freely changed, and the appearance of the print medium can be confirmed along with the image on the print medium. When the user operates a pointing device (not shown) on the image displayed on the image display unit 151, the image processing device 100 repeats a series of processes in which the rendering execution unit 121 performs rendering processing again and displays the processing result on the image display unit 151. Here, the pointing device may be a 3D mouse or a tracking ball, or may be a type of device that allows a multi-touch panel provided on the image display unit 151 to be operated with a finger or a touch pen. For example, when a multi-touch panel is provided on the surface of the image display unit 151, the print medium PLb and the light source LG may be moved directly with a finger or the like, or two fingers may be used to rotate the print medium PLb or change the distance between the light source LG and the print medium PLb in three dimensions.

Whenever the position or angle of the print medium PLb or the light source LG in this virtual space is changed, the rendering execution unit 121 performs a rendering process each time, and displays the rendered result RRD on the image display unit 151. An example of such a display is shown in FIG. 8. As shown in the figure, whenever the position or angle of the print medium PLb or the light source LG in the virtual space is changed, the print medium on which the image is printed is rendered based on physics each time, and is displayed in a manner close to how the actual print medium on which the image is printed appears in real space.

In particular, in this embodiment, the color of the image printed on the printing medium is converted to the color of the image that is actually printed by a color management system (CMS). In addition, in the lighting process during rendering,
[1] The print medium on which the image is printed is treated as a 3D object;
[2] The texture parameter TXT is used to take into account the texture of the surface of the printing medium;
[3] Furthermore, when considering the texture of the surface of the printing medium, the pixel values of the image data to be printed are taken into consideration;
Therefore, the reproducibility of the print medium displayed on the image display unit 151 is extremely high. The following describes the steps [1], [2], and [3].

Regarding [1]:
How a 3D object looks in a virtual space can be expressed using the bidirectional reflectance distribution function (BRDF) of reflected light at each part of the object and luminance. The bidirectional reflectance distribution function BRDF indicates the angular distribution characteristics of reflected light when light is incident from a certain angle. Luminance is the brightness of the object. Together, these are also called the illumination model. An example of the reflection model adopted in this embodiment is shown below. The BRDF can be expressed as a function f(x, ωl, ωv), and the luminance can be expressed as a function L(x, ωv), as shown in the following formulas (1) and (2).
f(x, ωl, ωv) = kD/π+kS*(F*D*V) ... (1)
L(x, ωv) = f(x, ωl, ωv) * E⊥(x) * n · ωl ... (2)
x: coordinates in the plane, ωv: viewpoint direction vector, ωl: light source direction vector, kD: diffuse reflectance (Diffuse Albedo), kS: specular reflectance (Specular Albedo)
F: Fresnel term, D: normal distribution function, V: geometric attenuation term, E⊥(x): illuminance incident perpendicularly to coordinate x, n: normal vector

The first term of the BRDF, kD/π, is a diffuse reflection component and is a Lambertian model. The second term is a specular reflection component and is a Cook-Torrance model. In formula (1), kd/π is sometimes called the diffuse reflection term, and kS*(F*D*V) is sometimes called the specular reflection term. The Fresnel term F, normal distribution function D, and geometric attenuation term V are known models and calculation methods, so their explanations are omitted. As the BRDF, a function according to the reflection characteristics of the 3D object surface and the purpose of rendering may be used. For example, the Disney Principled BRDF may be used. In this embodiment, the BRDF is used as a function representing the reflection of light, but the bidirectional scattering surface reflectance distribution function (BSSRDF) may also be used as a function representing the reflection of light.

As can be seen from the above formulas (1) and (2), the calculation of the reflection model requires a normal vector n, a light source direction vector ωl, and a viewpoint direction vector ωv. The print medium is treated as a 3D object composed of multiple tiny polygons as the subject of the rendering process, and the normal vector n, which reflects the minute unevenness of the print medium surface, is calculated from the polygon normal Np and a normal map described below. Therefore, the vertex pipeline VPL calculates the polygon normal Np and the UV coordinates that determine the reference position of the normal map, and inputs the light source direction vector ωl and the viewpoint direction vector ωv to the pixel pipeline PPL. In the pixel pipeline PPL, the pixel shader PS uses the UV coordinates to reference the normal map provided as one of the texture parameters, and calculates the normal vector n from the value of the referenced normal map and the polygon normal Np.

In this embodiment, as described above, the print medium on which the image is printed is treated as a 3D object, and physically based rendering is performed using the above formulas (1) and (2). Note that the light source direction vector ωl and the viewpoint direction vector ωv are calculated each time the user uses a pointing device to change the position and angle of the print medium PLb and the light source LG in the virtual space, as shown in FIG. 8.

Regarding [2]:
In this embodiment, the texture parameter TXT is used to take into account the texture of the surface of the print medium. The texture parameter TXT may be any of the following, but it is not necessary to take all of them into account. It is sufficient to take into account at least one of the parameters listed below, such as smoothness.
Smoothness (S) or roughness (R):
A parameter indicating the smoothness of the surface of a 3D object. Smoothness S is generally specified in the range of 0.0 to 1.0. Smoothness S affects the normal distribution function D and geometric attenuation term V of the BRDF in the above-mentioned formula (1). When this value is large, specular reflection becomes stronger, and a glossy appearance is presented. Roughness R may be used instead of smoothness S. The two can be converted as S = 1.0 - R. Note that smoothness is sometimes called smoothness and roughness is sometimes called roughness.

Metallic M (metallic):
Indicates the degree to which the surface of a 3D object is metallic. When the surface is highly metallic, the value of metallicity M is large. When metallicity M is large, the surface of the object is more likely to reflect light from the surroundings, resulting in a reflection of the surrounding scenery, making it easier for the color of the object itself to be hidden. Metallicity M affects the Fresnel term F.
The Fresnel term F can be expressed by the following equation (3) using the Schlick approximation.
F(ωl, h)=F ₀ +(1−F ₀ )(1−ωl·h) ⁵ ... (3)
Here, h is a half vector of the viewing direction vector ωv and the light source direction vector ωl, and F ₀ is the specular reflectance at normal incidence. The specular reflectance F ₀ can be directly specified as the color of the specular reflected light (specularColor), or it can be given by linear interpolation (here, represented as a lerp function) using metallicity M as shown in Equation (4).
F ₀ = lerp (0.04, tC, M) ... (4)
Here, tC is the texture color (albedoColor) of the 3D object. Note that the value 0.04 in formula (4) is a representative value for each of RGB, which indicates a general value for non-metals. The same is true for the texture color tC.

・Normal Map:
The normal map represents the normal vectors of the minute unevenness of the surface of the printing medium. By associating (pasting) the normal map with the 3D object, the normal vectors of the minute unevenness of the surface of the printing medium can be imparted to the 3D object. The normal map can affect the Fresnel term F, the normal distribution function D, and the geometric attenuation term V of the BRDF.

Other texture parameters:
Other parameters that can function as texture parameters include specular color, and clear coat layer parameters that indicate the presence or absence of a clear coat layer on the surface of the print medium, its thickness, transparency, and the like.

Regarding [3]:
Furthermore, in this embodiment, the same value is not used for the entire printing medium, particularly for the smoothness S of the texture parameter TXT, but the texture setting unit 131 modifies the smoothness S of the texture parameter TXT to a value smaller than the initial value S1 of the smoothness of the paper-white portion by using the gain g for pixels in the managed image data MGP whose pixel value is not zero, i.e., pixels that are not paper-white, assuming that ink is discharged. Therefore, even if light from the light source LG hits the same area, the appearance differs between a portion where ink is not discharged and a portion where ink is discharged. An example of this is shown in FIG. 9. The upper part of the figure shows the rendering result when the pixel value is not taken into consideration as the texture (here, smoothness) and the smoothness is set to the initial value S1 over the entire printing medium. The highlight portion HLt where the light from the light source LG is reflected is rendered with the same width in both the paper-white portion CN and the portion CI where ink is discharged. In addition, in both the upper and lower parts of the figure, the paper-white portion CN is slightly shaded so that it is easy to distinguish it from the highlight portion HLt.

The lower part of Figure 9 shows the rendering result when pixel values are taken into consideration as the parameter S indicating smoothness. In this case, the smoothness of the paper-white portion of the printing medium is set to an initial value S1, and the smoothness of the portion where ink is ejected is set to be smaller than the initial value S1. Therefore, the highlight portion HLG where light from the light source LG is reflected is rendered wider in the portion CI where ink is ejected than in the paper-white portion CN. This is because in the portion where ink is ejected, the smoothness S is smaller than the smoothness of the paper-white portion (initial value S1), and scattering of the light from the light source LG does not occur sharply.

As explained above,
[1] Treat the print medium on which the image is printed as a 3D object;
[2] Using the texture parameter TXT, the texture of the printing medium is taken into account;
[3] Furthermore, when considering texture, pixel values on the printing medium are taken into account.
As a result, the image processing device 100 of this embodiment can display the appearance of the print medium on which the image is printed on the image display unit 151 with a high degree of freedom and high reproducibility. As shown in FIG. 6, when the print medium is viewed from a direction facing the print medium, the texture of the print medium surface and the roughness caused by the fine unevenness of the print medium surface appear, and as shown in FIG. 8, when the print medium is rotated and viewed from an oblique direction, the illumination by the light source LG is reflected on the surface of the print medium, and the resulting highlight portion HLT appears. Furthermore, even on the surface of the same print medium, there is a difference in surface smoothness between the part where the ink is ejected and the paper white part where the ink is not ejected, so the highlight portion due to the reflection of the illumination by the light source LG is in a different state. Usually, in the part where the ink is ejected, the surface smoothness decreases and the highlight portion spreads. Note that the illumination light does not need to be limited to illumination directly directed at the print medium, such as a spotlight, and also includes sunlight, indirect illumination, and indirect light.

(A6) Modified Example:
Next, a modified example of the first embodiment will be described. In the image processing device 100 of the modified example, the process shown in Fig. 10 is performed instead of step S165 of the texture setting process shown in Fig. 4. The other processes are the same as those in the first embodiment. In this modified example, the smoothness correction value S is calculated as follows, and written to the alpha channel α-ch of the managed image data MGP.

After reading out a pixel value from one pixel of the managed image data MGP (step S164), in step S166, which is provided in place of step S165 in the first embodiment, a luminance value Y is first calculated from the pixel value (step S166a). The luminance value Y can be calculated from the sRGB color value of each pixel of the managed image data MGP by the following formula (5).
Y = 0.299 × R + 0.587 × G + 0.114 × B ... (5)
Furthermore, a correction coefficient coef is calculated from this luminance Y (step S166b). The correction coefficient coef is calculated by the following equation (6): where power() is a well-known function for calculating a power.
coef = power (Y, 0.25) ... (6)

After the correction coefficient coef is thus determined, a smoothness modification value S is determined by the following equation (7) (step S166c).
S = S1 × coef ... (7)
Here, S1 is the initial value of smoothness that is set for the entire print medium described in the first embodiment. The smoothness correction value S is obtained by the above processing, and then this is written to the alpha channel α-ch of the managed image data MGP (step S168).

According to the above-described modified example of the first embodiment, the same effect as that of the first embodiment is achieved, and the smoothness of each pixel is corrected according to the luminance corresponding to the amount of ink in that pixel to obtain the correction value S, so that the texture of the image can be expressed more accurately. FIG. 11 shows an example of a rendering result in the modified example. The example shown in the upper part of the figure is an example of how it looks when the pixel value is not considered as the texture (smoothness), and the example shown in the lower part of the figure is an example of how it looks when the pixel value, especially the luminance Y, is considered as the texture (smoothness). Normally, if the amount of ink ejected to one pixel increases, the luminance Y decreases, so if the luminance is low, the amount of ink ejected is large, the smoothness of that pixel decreases, and the range of the highlight part becomes wider. In the figure, it can be seen that the range of the highlight part, especially the width, differs in the parts with different luminance in the multiple areas CK, CB, CR, and CY painted in different colors. In addition, the width of the highlight part of the part CI where ink is ejected, which is composed of these multiple areas CK, CB, CR, and CY, is wider than the width of the highlight part of the paper white part. In this way, by using pixel values and corresponding brightness values to modify smoothness, which is one of the texture parameters, it is possible to more accurately represent how the actual print medium on which the image is printed will look.

In the first embodiment described above, the texture parameter TXT is mainly set or modified according to the pixel value of the managed image data MGP, mainly to the smoothness of the printing range. As described above, such texture parameters TXT may include not only the smoothness or roughness of the surface of the printing medium, but also various parameters such as metallicity, normal map, specular color, clear coat layer parameters indicating the presence or absence of a clear coat layer on the surface of the printing medium, its thickness, or transparency. All of these may be handled, or only some of the parameters may be handled. Of course, one of these texture parameters TXT may be handled. In addition, among the texture parameters TXT, some may be significantly affected by whether an image is formed or the number of pixel values, while others may be less affected or not affected at all. In such a case, the one that takes a different value depending on the pixel value may be the first texture parameter, and the one that is not affected or is less affected by the pixel value may be the second texture parameter, and the first texture parameter may be corrected according to the pixel value. Of course, the second texture parameter may also be corrected according to the pixel value. For example, among the texture parameters, the normal map is not affected by pixel values, or is less affected by them, so it may be possible to treat it as the second texture parameter. This also applies to the second and subsequent embodiments described below.

B. Second embodiment:
The image processing device 100 of the second embodiment has a hardware configuration similar to that of the first embodiment, but differs from the first embodiment in that a texture setting unit 131A for setting texture is provided in the rendering execution unit 121. As shown in FIG. 12, the image processing device 100 of the second embodiment has a texture setting unit 131A provided inside the pixel shader PS. The texture setting unit 131A inputs managed image data MGP and texture parameters TXT, and performs the texture setting process shown in FIG. 4 of the first embodiment and FIG. 10 showing the main part in its modified example based on these. In this case, it is not necessary to write the calculated smoothness correction value S in the alpha channel α-ch of the managed image data MGP, and it is sufficient to directly use it in the calculation process of texture expression in the pixel shader PS. With this configuration in which the texture setting unit 131A is provided inside the pixel shader PS, it is possible to shorten the time from reading the image to be rendered to the first display of the rendering result. Note that, in such a second embodiment, the same effects as those of the first embodiment can be obtained.

C. Third embodiment:
Next, a third embodiment of the image processing device 100 will be described. The image processing device 100 of the third embodiment has a configuration similar to that of the first embodiment, and instead of the texture setting unit 131 of the first embodiment, has a texture setting unit 131B shown in FIG. 13. The texture setting unit 131B inputs the device image data DCD and the printing conditions PCC acquired from the CMS 111, and calculates the texture parameter TXT. As in the first embodiment, the calculated texture parameter TXT may be written into the alpha channel α-ch of the managed image data MGP and input to the pixel pipeline PPL of the rendering execution unit 121, or may be input to the pixel pipeline PPL in the form of a map image associated with the managed image data MGP.

As shown in the figure, the texture setting unit 131B includes a texture table acquisition unit 133 and a texture parameter calculation unit 134. The texture table acquisition unit 133 acquires a texture table according to the printing conditions PCC. The parameter calculation unit 134 calculates a texture parameter TXT, which is a smoothness correction value S in this case, according to the pixel value of the managed image data MGP, from the texture table TTB acquired by the texture table acquisition unit 133 and the device image data DCD acquired from the CMS 111. The device color image data DCD is image data that expresses the image data ORG in the color system of the printing device, which is the device that actually performs printing. For example, when the input color system of the printing device is CMYK, the image data ORG is converted by the CMS 111 into the CMYK color system using an output profile compatible with the printing device, and the image data is input to the texture setting unit 131B as the device color image data DCD.

The smoothness correction value S is calculated as follows.
The printing conditions PCC are received, and a texture table TTB corresponding to the printing conditions PCC is obtained. Textures, such as surface smoothness and metallicity, vary depending on the printing medium and also on the ink used for printing. When information on the printing device and ink set is input as the printing conditions PCC, the texture table acquisition unit 133 acquires the corresponding texture table TTB from a number of texture tables prepared in advance. An example of such a texture table TTB is shown in FIG. 14. In this example, the texture table TTB defines the texture parameter TXT in the case of an ink system, in this case the density of each color value of CMYK, as an input. The texture parameter TXT defines the surface smoothness S of the printing medium, metallicity M, and specular reflection components s_R, s_G, and s_B of each color of RGB.

For ease of understanding, in FIG. 14, the first row shows the case of no color (0% for all CMYK), the second row shows the case of 100% yellow ink Y only, and the third row shows the case of 100% black ink only. The actual texture table TTB is saved in a format that shows the texture parameter TXT when each of CMYK is changed in 10% increments, for example. Since there is an ink duty limit for the amount and total of each ink, it is sufficient to create a table that shows the value of the texture parameter TXT for the combination of CMYK inks within the ink duty limit. The proportion of each ink may be changed in finer increments, for example, 1%, or in coarser increments. Furthermore, the increments by which the proportion of each ink is changed do not need to be uniform, and may be different for each ink, and the degree to which the ink proportion is changed may differ depending on the density.

In the illustrated texture table TTB, if the pixel value is 0 (colorless) for all CMYK, it corresponds to the white part of the paper, and as in the first embodiment, the smoothness S is 0.95 and the metallicity M is 0. The specular reflection components s_R, s_G, and s_B of each of the RGB colors are all 1. On the other hand, as shown in the second row, when only the yellow ink Y is 100%, the smoothness S is set to a value of 0.92 and the metallicity M is set to a value of about 0.2 rather than 0 due to the presence of the ink on the surface of the printing medium. This value corresponds to the bronzing phenomenon that can occur due to the yellow ink Y. When the yellow ink Y is present in high concentration, the surface of the printing medium may appear shiny with a bronze color. The specular reflection components s_R, s_G, and s_B of each of the RGB colors are set to values corresponding to the bronze color, as shown in the figure. The example shown in the third row shows the case where the black ink K is 100%. In the case of black ink K, the surface smoothness S is set to about 0.8, which is lower than the other inks. Also, since it is black, most of the light is absorbed, so the specular reflection components s_R, s_G, and s_B of each color of RGB are all set to a value of almost 0. The texture values in the texture table TTB vary depending on the printing device and ink set, so there are cases where the number of texture values is required is equal to the number of printing devices x ink sets. Also, if there are printing modes with different amounts of ink ejected as printing conditions PCC, such as "draft printing" and "high-definition printing," it is desirable to prepare only those combinations. Of course, it is also acceptable to reduce the number of types of texture table TTB in order to find combinations that produce similar textures. All texture tables TTB may be stored in advance in 131B, but those that are used less frequently may be stored on an external site and, when necessary, may be read from the external site via a network or the like.

The parameter calculation unit 134 refers to the texture table TTB acquired by the texture table acquisition unit 133 according to each pixel value of the device color image data DCD input from the CMS 111, acquires texture parameters TXT such as smoothness S and metallicity M, and outputs these to the pixel pipeline PPL together with the managed image data MGP. At this time, the texture parameters TXT may be stored in the alpha channel α-ch of the managed image data MGP as in the first embodiment and output to the pixel pipeline PPL, or if the texture parameters TXT include multiple parameters such as smoothness S and metallicity M, they may be stored in a single file as a map image and output to the pixel pipeline PPL.

The image processing device 100 of the third embodiment described above has the same effect as the first embodiment, and since the texture parameter TXT is calculated using the texture table TTB, it is possible to prepare an appropriate texture parameter TXT for each pixel value of the managed image data MGP under a variety of printing conditions PCC. In addition, it is easy to adjust the texture parameter TXT for each printing condition PCC.

D. Fourth embodiment:
The fourth embodiment is in the form of a printing system 300. As shown in FIG. 15, the printing system 300 includes the image processing device 100 shown in the first to third embodiments, an image preparation device 310 including a printing condition specification unit 315, and a printing device 320. In this embodiment, the image preparation device 310 is a computer used by a user, and is a device that prepares image data ORG to be printed. The image preparation device 310 may have a function of creating an image, or may simply store image data and provide it to the image processing device 100 as necessary. The image preparation device 310 is connected via a network NW so that the image processing device 100 can obtain image data ORG, but may also be directly connected to the image processing device 100 by wire or wirelessly.

In this embodiment, the printing device 320 is connected to the image preparation device 310 via the network NW, receives instructions from the image preparation device 310, and prints the image data ORG output by the image preparation device 310 onto the printing medium PRM. Prior to printing by the printing device 320, the user of the printing system 300 causes the image processing device 100 to acquire the image data ORG, and as described in the first to third embodiments, treats the printing medium PRM as a 3D object and performs lighting processing using texture parameters to render the printing medium PRM including the image data printed thereon.

The printing condition specification unit 315 provided in the image preparation device 310 specifies printing conditions that affect the appearance of the image printed on the printing medium on the printing medium. The printing condition specification unit 315 may be provided in the image processing device 100. The printing condition specification unit 315 sets a profile required for color conversion from printing conditions such as the selection of a paper tray containing a specific printing medium, the selection of an ink set to be used, the selection of the type of printing device to be used, the printing resolution, and the printing quality, and determines a texture parameter TXT to be referenced based on the printing conditions. Since the image processing device 100 performs rendering using this texture parameter, it is possible to easily check the appearance of the printing medium on which the image is printed, reflecting differences in printing conditions. In addition to these conditions, the printing condition specification unit 315 may set the observation state of the printing medium on which the image is printed in the virtual space, lighting information which is information on the lighting for the printing medium in the virtual space, object specification information which specifies a 3D object in the virtual space, background information which specifies the background in the virtual space, and the like.

The texture parameter TXT may be set by any of the methods of the first to third embodiments, but FIG. 15 illustrates a method using the texture table TTB, specifically the method described as the third embodiment. This method requires a texture table TTB for determining the texture parameter TXT from the printing conditions. In the figure, the texture table TTB corresponding to the ink sets and printing media in various printing devices is stored on an external site 200 connected via a network NW, and the printing condition specification unit 315 of the image preparation device 310 sets the texture parameter TXT by referring to the texture table TTB, and outputs this to a pixel pipeline of a rendering execution unit (not shown) of the image processing device 100, where rendering processing is performed.

The user checks the rendering result by the image processing device 100 on the image display unit 151, and if necessary, changes the viewpoint or light source position, or the light source intensity or white balance, etc. to check how the print medium PRM looks, and then outputs the image data ORG from the image preparation device 310 to the printing device 320 via the network NW to print the image data ORG on the print medium PRM. Prior to printing, the user can check how the image looks on the print medium PRM by the physically based rendering by the image processing device 100. As a result, the user can check the difference in texture due to the type of print medium PRM, including the smoothness (roughness) of the surface of the print medium PRM, before printing. It is also possible to change the color of the image data ORG, change the type of print medium PRM used, change the printing device 320 used for printing, or change the ink set, so as to obtain the desired print result by looking at the rendering result displayed on the image display unit 151.

The printing medium on which the printing device 320 prints may be something other than paper. For example, it may be a textile printer that prints on fabric, or a printing device that prints on solid objects such as cans and bottles. In addition to a configuration that prints directly on an object, a printing device configuration that prints on a transfer medium such as transfer paper and transfers the ink formed on the transfer medium to the fabric or solid object that is the printing medium can also be adopted. A dye-sublimation printing device is one such transfer type printing device. In such a transfer type configuration, the printing medium is the final printed matter to which the ink is transferred. In such a case, texture parameters related to the structure and texture of the surface of the printing medium such as fabric, metal, glass, or plastic are prepared according to the properties of the printing medium, and physically based rendering can be performed by the image processing device 100. In a transfer type printing device, the texture parameters used represent the texture of the final printed matter, not the transfer medium. An example of a display on the image display unit 151 when printing on such fabric or a can is shown in FIG. 16. For ease of understanding, the figure shows both an object OBJt printed on a T-shirt and an object OBJc printed on a can, but typically each print medium is displayed one at a time. Of course, multiple rendering execution units may be prepared, and multiple physically based rendering results may be displayed simultaneously.

E. Other embodiments:
(1) The present disclosure may also be implemented in the following forms. One of the embodiments is an image processing device. The image processing device includes a printing condition specification unit that specifies printing conditions including a type of printing medium, an image input unit that inputs printing image data including pixel values of each pixel of a printing image formed on the printing medium, a texture setting unit that sets texture parameters representing the texture of the printing medium for which the printing conditions have been specified, taking into account the pixel values of the printing image data at the location where the printing image is formed, and a rendering execution unit that generates a rendering image of the printing medium on which the printing image has been formed by a rendering process using the printing image data and the texture parameters.

In this way, it is possible to reproduce how the print medium printed based on image data looks, taking into consideration the pixel values of the image formed on the print medium, particularly with regard to the texture of the print medium. In this image processing device, the print medium is rendered as if an image were printed on it, and a rendering image of the print medium on which the print image is formed is generated, so that the appearance of the print medium on which the image is printed can be reproduced with high precision. In particular, it is possible to reproduce the phenomenon in which different textures appear in the image-formed area and the image-non-formed area, and a more realistic rendering image can be displayed. Therefore, it is possible to show and confirm to the user that there are parts in the image with different textures without actually printing them. In addition, since the formation of such a rendering image is performed using texture parameters in the rendering process rather than editing the original image, it is also possible to reflect and confirm the effects of the light source and viewpoint in real time. This makes it possible to confirm the appearance of the print medium on which the image is formed, including the color, prior to printing, and suppresses discrepancies in the impression when the image to be printed is combined with the print medium, and it is possible to reduce the need to adjust the original image and printing conditions and repeat trial and error, thereby reducing the cost and time required for trial printing.

Such an image processing device may be configured as a device that performs only the above image processing, or as a device that also includes a function for saving images to be printed. Alternatively, it may be configured as a device that includes a function for creating images to be printed, or as a device that prints images. The image processing device may be realized by a computer equipped with a GPU, or may be configured as a distributed system in which the necessary functions are located at multiple sites and can be linked together. When configured as a distributed system, the processing load on the terminals is reduced, making it easier to perform the above image processing even on mobile terminals such as tablets, further improving convenience for users.

The pixel values handled by the image processing device can take values from 0 to 255 if they are 8-bit integers, but they are not limited to 8 bits and can be expressed in 12-bit or other formats. Alternatively, they can be handled as values in the range of 0 to 100% or 0 to 1.0.

The rendering execution unit in such an image processing device can employ various existing configurations. In general, rendering may be performed by dividing it into multiple elements, such as viewpoint transformation that converts three-dimensional world coordinates into a coordinate system seen from the viewpoint, culling that removes vertices unnecessary for rendering from the 3D object, clipping that removes invisible coordinates, and rasterization. These processes may be realized by a configuration suitable for processing on a dedicated GPU, and a pipeline configuration that includes a vertex pipeline that processes the vertices of the 3D object and a pixel pipeline that processes each rasterized pixel.

The texture parameters handled by the rendering execution unit include the smoothness or roughness of the print medium's surface, metallicity, normal map, specular color, and clear coat layer parameters indicating the presence or absence of a clear coat layer on the print medium surface, its thickness, or transparency. Any other element that affects the appearance of the print medium may be treated as a texture parameter.

When dealing with these various texture parameters, the considerations when setting texture parameters that represent the texture of a print medium with specified printing conditions, taking into account the pixel values of the print image data at the location where the print image is formed, differ depending on the recording method used to form the image and the recording material used to form the image. Image recording methods include not only inkjet printers, but also laser printers and dye sublimation printers (direct type, transfer type), and there are various recording materials, such as dyes and pigments ejected with a solvent, toners that are thermally fused, and solid dyes. Therefore, when setting texture parameters, pixel values should be taken into account according to these differences.

(2) In the above configuration, the texture setting unit may store texture parameters in advance in association with the printing conditions of the printing medium and pixel values that the print image data can take, and may set the texture parameters at the formation location of the print image according to the printing conditions and the pixel values of the print image formed on the printing medium. In this way, the texture parameters can be easily set from the printing conditions of the printing medium and the pixel values of the print image data. Therefore, even if the printing conditions that affect the appearance of the image are different, it is easy to respond. Such responses may be made in detail for each printing condition, or a basic relationship may be stored in advance, and minor differences in the printing conditions may be responded to by correcting this basic relationship. When it is important to check the difference in appearance due to the pixel values of the print image data, correction or modification due to such minor differences in the printing conditions may not be made. In this specification, "correction" includes not only a process of setting a parameter to its original value, but also a parameter modification process that brings the parameter closer to its original value to a certain extent.

(3) In the configuration of (1) or (2) above, the texture setting unit may store a table that associates the texture parameters with the printing conditions of the printing medium and the pixel values that the print image data can take, and may set the texture parameters at the location where the print image is formed by referencing the table. In this way, it is possible to easily handle complex correspondences, since it is only necessary to store and refer to the table. Of course, it is also acceptable to configure the texture parameters to be calculated based on pixel values for each printing condition.

(4) In the configurations (1) to (3) above, the texture setting unit may store the table in association with the printing conditions, and select the table to be referenced according to the specified printing conditions. In this way, even if multiple correspondences exist, they can be handled simply by selecting a table. It is also possible to store one or multiple tables showing basic relationships, and determine the differences between multiple printing conditions and pixel values by correction using the values stored in the tables.

(5) In the configurations (1) to (4) above, the texture setting unit may set the texture parameter by correcting a texture parameter previously set for the print medium in accordance with the print image data at the location where the print image is formed. In this way, the texture parameter can be corrected in accordance with the print image data, and the rendering image can be displayed in an appearance that corresponds to the print image data. Correction in accordance with the print image data includes, for example, a case where the correction content is different between a portion of the print image data where an image is formed and a portion of the print image data where an image is not formed. Furthermore, the correction is not limited to a correction in accordance with the pixel value of each pixel, and a configuration may be adopted in which the texture parameter is set using, for example, an average pixel value of the print image data.

(6) In the above configurations (1) to (5), the correction of the texture parameter may be performed using a pixel value included in the print image data. In this way, the rendering image can be displayed in an appearance according to the pixel value. When the pixel value represents the amount of ink, the correction using the pixel value is such that the greater the amount of ink, the smaller the smoothness of the texture parameter is, for example. When the pixel value represents lightness or luminance, the higher the pixel value, the smaller the rate at which smoothness is reduced is the correction. When the texture parameter is metallic, the correction is such that the value of metallicity is determined according to the density value of ink of a specific color. Note that even when an image is formed using ink, the degree of correction of the texture parameter using the pixel value differs between when an image is formed using a thermal dye-sublimation ink and when an inkjet printer or the like ejects a dye or pigment together with a solvent. When a laser printer or the like is used to thermally fuse toner, the manner of correction of the texture parameter with respect to the pixel value is also different. The manner of correction may be determined according to the properties of the recording material used in each image formation.

(7) In the configurations (1) to (6) above, the correction may not be performed in any area other than where the print image is formed. This can simplify the correction process.

(8) In the configurations (1) to (7) above, the texture setting unit may perform the correction using the luminance or brightness of pixels constituting the print image as the pixel value. This simplifies the process of correcting texture parameters when forming an image using a multi-color printing device or a printing device that uses multiple recording materials such as inks even for a single color.

(9) In the above configurations (1) to (8), the pixel value may include at least one of the saturation and hue of the pixel in addition to the luminance or brightness. In this way, more precise correction can be performed than when it is simply based on luminance or brightness. For example, by appropriately taking into account saturation and hue, it is possible to express differences in appearance due to phenomena that can occur in a specific color range, such as the bronzing phenomenon that can occur in inkjet printers, etc.

(10) In the configurations (1) to (9) above, the texture setting unit may store correction parameters corresponding to the pixel values for each type of printing medium, obtain the correction parameters for each pixel according to the specified printing conditions, and perform the correction using the obtained correction parameters. In this way, the texture parameters can be easily corrected based on the printing conditions.

(11) In the configurations (1) to (10) above, the texture setting unit may use, as the texture parameter, a texture parameter previously set for the printing medium for which the printing conditions are specified, except for the location where the print image is formed. This can simplify the processing.

(12) In the configurations (1) to (11) above, the texture setting unit may set the texture parameter using a first texture parameter that takes a different value depending on the pixel value and a second texture parameter that is not affected by the pixel value, and set the first texture parameter in the area where the print image is formed, taking into account the pixel value of the print image data. In this way, it is necessary to take into account pixel values for only some of the texture parameters, simplifying the processing.

(13) Another embodiment is a printing system. This printing system includes the image processing device according to any one of (1) to (12) above, a display unit that displays the rendering image generated by the image processing device, and a printing device that prints the print image data on the printing medium. In this way, when printing is performed by the printing device, the appearance of the printing medium on which the image is printed is displayed on the display unit prior to printing, so that it is possible to check this before printing. Therefore, it is possible to prevent discrepancies between the image to be printed and the impression of the printing medium, and it is possible to reduce the need to adjust the original image or printing conditions and repeat trial and error, thereby reducing the cost and time required for printing trials. Note that the image device alone may also be provided with a display unit that displays the rendering image. In this way, the user can immediately check the rendering result using the display unit. In addition, if a tablet or the like is used as the display unit, it is possible to work away from the image processing device and the printing device that performs printing.

(14) Yet another embodiment is an image processing program that generates a rendering image of a print medium on which an image is printed. This image processing program may have a first function of specifying printing conditions including the type of the print medium, a second function of inputting print image data including pixel values of each pixel of the print image formed on the print medium, a third function of setting texture parameters representing the texture of the print medium for which the printing conditions have been specified, taking into account the pixel values of the print image data at the location where the print image is formed, and a fourth function of generating a rendering image of the print medium on which the print image is formed by rendering processing using the print image data and the texture parameters. In this way, the image processing device described as (1) can be easily configured in an apparatus equipped with a computer.

(15) In each of the above embodiments, a part of the configuration realized by hardware may be replaced by software. Alternatively, at least a part of the configuration realized by software may be realized as a hardware configuration, for example, by a discrete circuit configuration. In addition, when a part or all of the functions of the present disclosure are realized by software, the software (computer program) may be provided in a form stored in a computer-readable recording medium. The term "computer-readable recording medium" is not limited to portable recording media such as floppy disks and CD-ROMs, but also includes internal storage devices within a computer, such as various RAMs and ROMs, and external storage devices fixed to a computer, such as hard disks. In other words, the term "computer-readable recording medium" has a broad meaning that includes any recording medium to which data packets can be fixed, not temporarily.

The present disclosure is not limited to the above-described embodiments, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each form described in the Summary of the Invention column can be replaced or combined as appropriate to solve some or all of the above-described problems or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate.

100...image processing device, 111...color management system, 121...rendering execution unit, 125...rendering condition setting unit, 131, 131A, 131B...texture setting unit, 132...correction value calculation unit, 133...texture table acquisition unit, 134...texture parameter calculation unit, 135...printing condition specification unit, 139...image memory, 151...image display unit, 171...storage unit, 200...site, 300...printing system, 310...image preparation device, 315...printing condition specification unit, 320...printing device

Claims (15)

a printing condition specification unit that specifies printing conditions including a type of printing medium;
an image input unit for inputting print image data including pixel values of each pixel of a print image to be formed on the print medium;
a texture setting unit that sets a texture parameter representing a texture of the printing medium for which the printing conditions have been specified, in consideration of the pixel value of the print image data at a location where the print image is to be formed;
a rendering execution unit that generates a rendering image of the printing medium on which the print image is formed by a rendering process using the print image data and the texture parameters;
An image processing device comprising: The texture setting unit is
texture parameters are stored in advance in association with the printing conditions of the printing medium and pixel values that the print image data can take;
setting the texture parameter at the portion where the print image is formed in accordance with the printing conditions and the pixel value of the print image formed on the print medium;
The image processing device according to claim 1 . The texture setting unit is
a table is stored in which the texture parameters are associated with the printing conditions of the printing medium and pixel values that the print image data can take;
By referring to the table, the texture parameter is set at the portion where the print image is formed.
The image processing device according to claim 2 . The texture setting unit is
The table is stored in association with the printing conditions,
selecting the table to be referenced in accordance with the specified printing conditions;
The image processing device according to claim 3 . The image processing device according to claim 1, wherein the texture setting unit sets the texture parameters by correcting texture parameters previously set for the print medium in accordance with the print image data at the location where the print image is formed. The image processing device according to claim 5, wherein the correction of the texture parameters is performed using pixel values contained in the print image data. The image processing device according to claim 6, wherein the correction is not performed in any area other than where the print image is formed. The image processing device according to claim 6, wherein the texture setting unit performs the correction using the luminance or brightness of the pixels constituting the print image as the pixel value. The image processing device according to claim 8, wherein the pixel value includes at least one of the saturation and hue of the pixel in addition to the luminance or the brightness. The texture setting unit is
a correction parameter corresponding to the pixel value is stored for each type of the printing medium;
acquiring the correction parameters for each pixel according to the specified printing conditions;
performing the correction using the acquired correction parameters;
The image processing device according to claim 6. The image processing device according to claim 1, wherein the texture setting unit uses, as the texture parameter, a texture parameter that is preset for the printing medium for which the printing conditions are specified, except for the portion where the print image is formed. The texture setting unit is
The texture parameter is set to a first texture parameter having a different value depending on the pixel value and a second texture parameter not affected by the pixel value;
At the location where the print image is formed, the first texture parameter is set in consideration of the pixel value of the print image data.
The image processing device according to claim 1 . The image processing device according to claim 1, further comprising a display unit for displaying the rendering image. An image processing device according to any one of claims 1 to 12,
a display unit that displays the rendering image generated by the image processing device;
a printing device that prints the print image data on the print medium;
A printing system comprising: An image processing program for generating a rendering image of a print medium on which an image is printed, comprising:
A first function for identifying printing conditions including a type of the printing medium;
a second function of inputting print image data including pixel values of each pixel of a print image to be formed on the print medium;
a third function of setting a texture parameter representing a texture of the printing medium for which the printing conditions have been specified, in consideration of the pixel value of the print image data at a location where the print image is to be formed;
a fourth function of generating a rendering image of the printing medium on which the print image is formed by a rendering process using the print image data and the texture parameters;
An image processing program that realizes this using a computer.

Images