JP6349355B2 - Image composition apparatus, information processing apparatus, and image composition method - Google Patents

Description

  The present invention relates to an image composition device that superimposes and displays a plurality of images, an information processing device including the same, and an image composition method.

  Conventionally, various techniques for improving the image quality in video display such as television broadcasting and distribution video have been developed. In recent years, in addition to techniques for improving resolution and color gamut, techniques for processing HDR (High Dynamic Range) signals with an expanded luminance range are becoming widespread. Compared to the conventional standard dynamic range (SDR), HDR has an allowable luminance range of about 100 times, so that objects that are dazzling in the real world, such as reflected sunlight, can be expressed more realistically on the image. can do. In the world of computer graphics, such as game images, as well as television broadcasts and distribution videos, the virtual world gives a sense of realism when expressed in HDR.

  Regardless of the type of electronic content, there are cases where additional information related to operations and display contents, such as a help screen, a control panel, and an indicator, is displayed while the main image is being displayed. In order to display such a secondary image, a basic function commonly provided by the apparatus for content is called and used in many cases. In this case, the specification differs from the main image representing the content main body, and the world view of the content can be greatly impaired by simply displaying it. When integrating with a main image, it is not efficient to prepare a secondary image generation mechanism according to each of a plurality of specifications such as SDR and HDR.

  The present invention has been made in view of these problems, and a purpose of the present invention is to provide a technique capable of suitably fusing and displaying separately generated images using a common generation mechanism even if the luminance range of the content image is different. It is to provide.

  One embodiment of the present invention relates to an image composition device. The image composition apparatus includes a first image data acquisition unit that acquires data of a first image, and a superposition that is obtained by multiplying each pixel value of a second image to be superimposed and displayed on the first image by an alpha value that indicates transparency. A superimposition data acquisition unit for acquiring the intermediate image data, a second image data acquisition unit for acquiring the second image data from the intermediate image, and the luminance value represented by the second image data by the first image data Data of a display image obtained by superimposing the second image on the first image by performing an alpha blending process using the data conversion unit that converts the value into a value in the luminance space to be represented and the converted second image data And a synthesis processing unit for generating and outputting the data.

  Another aspect of the present invention relates to an information processing apparatus. The information processing apparatus generates an image generation unit that executes electronic content and generates data of the first image, and generates data of the second image when it is necessary to display the second image superimposed on the first image. In addition, a superimposition data generation unit that generates data for a superimposition intermediate image obtained by multiplying each pixel value by an alpha value indicating transparency, the first image data and the intermediate image data are acquired, and the second image is acquired. An image composition unit that generates display image data superimposed on the first image, and an image output unit that acquires image data from either the image generation unit or the image composition unit and outputs the image data to the display device, The image composition unit converts a luminance value represented by the second image data into a value in the luminance space represented by the first image data, and a second image data obtaining unit that obtains the second image data from the intermediate image. Data converter and after conversion A synthesizing unit which generates data of a display image by performing the alpha blending process using the data of the second image, characterized by comprising a.

  Another embodiment of the present invention relates to an image composition method. In this image composition method, the image composition apparatus obtains the data of the first image, and superimposes obtained by multiplying each pixel value of the second image to be superimposed and displayed on the first image by an alpha value indicating transparency. The step of acquiring the data of the intermediate image, the step of acquiring the data of the second image from the intermediate image, and the luminance value represented by the data of the second image are converted into values in the luminance space represented by the data of the first image. And a step of generating alpha image processing using the converted second image data to generate display image data in which the second image is superimposed on the first image, and outputting the display image data to the display device; , Including.

  Note that any combination of the above-described components, and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a recording medium on which the computer program is recorded, and the like are also effective as an aspect of the present invention. .

  According to the present invention, even if the luminance range of the content image is different, it is possible to display an image that has been separately generated using a common generation mechanism and that is suitably fused.

It is a figure which shows the structural example of the information processing system in this Embodiment. It is a figure which shows typically the image which an information processing apparatus produces | generates in this Embodiment. It is a figure which shows the functional block of the image composition apparatus which synthesize | combines a 1st image and a 2nd image and produces | generates a display image. It is a figure which shows the internal circuit structure of the information processing apparatus in this Embodiment. It is a figure which shows the structure of the functional block of the information processing apparatus in this Embodiment. It is a figure which compares the brightness | luminance of a display image when the intermediate image is converted as it is and when the brightness component of the second image is acquired and converted according to the present embodiment. It is a flowchart which shows the process sequence which the information processing apparatus in this Embodiment produces | generates and outputs a display image.

  FIG. 1 shows a configuration example of an information processing system in the present embodiment. The information processing system includes an information processing device 10, an input device 14, and a display device 16. As illustrated, the information processing apparatus 10 may be connectable to a server or the like that provides various contents via a network 8 such as the Internet. The input device 14 detects general physical devices such as a controller, a keyboard, a mouse, a joystick, and a touch pad, an imaging device that captures the real world such as a user, a microphone that acquires sound, and various physical values. A sensor or any combination thereof may be used.

  The display device 16 is realized by a liquid crystal display, a plasma display, an organic EL display, or the like that displays an image. Further, a speaker that outputs sound may be provided. The input device 14 and the display device 16 may be connected to the information processing device 10 by a wired cable, or may be wirelessly connected by a wireless LAN (Local Area Network) or the like. Further, the external shapes of the input device 14, the display device 16, and the information processing device 10 are not limited to those illustrated in the drawings, and for example, two or more of them may be integrally formed.

  The information processing apparatus 10 receives a signal related to a user operation from the input device 14, performs processing according to the signal, generates display image data, and outputs the display image data to the display device 16. The information processing apparatus 10 may be a game machine, a personal computer, a tablet terminal, a mobile terminal, a mobile phone, or the like. The contents of the processing performed by the information processing apparatus 10 may vary depending on the form of the information processing apparatus 10 and the application selected by the user.

  For example, the information processing apparatus 10 advances an electronic game designated by the user according to a user operation, and generates and outputs data of the game screen at a predetermined frame rate. Alternatively, moving image data may be acquired from the server via the network 8, and sequentially decoded and output. As described above, the purpose of use of the information processing apparatus 10 may be various, and the content of information processing to be performed differs accordingly. Hereinafter, an explanation will be given with emphasis on a technique for displaying a plurality of images in a superimposed manner, such as an image generated as a result of such information processing or an image representing information to be presented.

  FIG. 2 schematically shows an image generated by the information processing apparatus 10 in the present embodiment. The image 200a is an image that is mainly displayed, such as a game or a moving image. The image 200b is an image that is temporarily displayed as necessary, and the illustrated example includes a dialog box that allows the user to input an address and a password for login. When such a dialog box needs to be displayed, the information processing apparatus 10 superimposes the image 200b including the dialog box on the image 200a that was originally displayed, and generates and outputs a display image 202.

  At this time, by making the image 200a visible through the image 200b in as wide a region as possible, necessary information can be suitably fused without interruption of the world view of the image 200a such as a game or a moving image. Furthermore, if the transparency of the image 200b is changed with time, it is possible to produce an effect that the dialog box gradually appears or disappears.

  It should be understood by those skilled in the art that there are various cases in which a plurality of images are superimposed and displayed in addition to the illustrated example. For example, in the case of a racing game, it may be possible to additionally display an image in which the entire course is viewed over the main image showing the driver's field of view. When displaying a movie, it may be possible to additionally display an image indicating bibliographic information such as a synopsis or a performer, or an operation panel such as playback, pause, or fast-forward. As described above, the overlapping position, area, and period may vary depending on the situation and the content to be displayed. Hereinafter, an image located behind the superimposed image 200a is referred to as a first image, and an image 200b located in front of the superimposed image 200b is referred to as a second image. However, even if three or more images are superimposed and displayed, the principle described below is the same.

  First, in order to clarify the features of the information processing apparatus according to this embodiment, an aspect in which the display image 202 is generated with a simpler configuration will be described. FIG. 3 illustrates functional blocks of an image composition apparatus that generates a display image by combining the first image and the second image. The image composition apparatus 300 is included in the information processing apparatus as shown in FIG. 1 and functions when the second image is superimposed and displayed according to a request from the user.

  The image composition device 300 includes a first image data acquisition unit 302 that acquires data of a first image, a superimposition data acquisition unit 304 that acquires data of a second image, and a second image superimposed on the first image. A combining unit 306 for combining is included. As described above, the first image is a main image of electronic content such as a game, and is separately generated by a function for executing an application or transmitted from an external server via the network 8.

For example, the second image is generated using a function provided by the information processing apparatus. The synthesizing unit 306 acquires the first image data and the second image data from the first image data acquisition unit 302 and the superimposed data acquisition unit 304, respectively, and generates and outputs a display image by combining them as shown in FIG. . In order to superimpose and display the second image with transparency as described above, the synthesis unit 306 determines the luminance F _out of each pixel of the display image by an alpha blend process represented by the following equation.
F _out = (1−α) Fb ₁ + αFb ₂ (Formula 1)

Here, Fb ₁ and Fb ₁ are the luminance of the pixel in the first image and the second image, respectively, and α is a general α value set for the pixel of the second image, that is, 0 or more indicating transparency. The value is 0 or less. For example, when the α value is changed from 0 to 1 in the entire image, the color of the second image gradually becomes darker from the state where only the first image is displayed, and the second image is finally displayed opaquely. Will be. If the α value is an intermediate value larger than 0 and smaller than 1, the second image becomes translucent with a density corresponding to the numerical value, and the first image can be seen through.

If the first image and the second image are RGB images, the luminances Fb ₁ and Fb ₂ are set for each of the three channels. In the present embodiment, they are collectively referred to as luminances Fb ₁ and Fb ₂ . Represent. Further, since the luminance values Fb ₁ , Fb ₂ and α value are set for each pixel, strictly speaking, although it depends on the two-dimensional position coordinates (x, y) on the image plane, Equation 1 relates to the pixels at the same position. Position coordinates are not shown because calculation is assumed. The following description is also the same.

Since the second image is generated with the intention of being superimposed on the first image from the beginning, the second image is provided as a set with an α value that determines how the second image is displayed. That is, the superimposition data acquisition unit 304 acquires the α value together with the luminance Fb _{2 of} each pixel, and supplies it to the synthesis unit 306. Here, considering that the second image and the α value are generated as a set, it is efficient to multiply the α value determined for each pixel by the luminance Fb ₂ represented as the pixel value of the second image. . That is, if the value αFb ₂ obtained by multiplying the α value is calculated at the stage where the second image is generated, the second term on the right side of Equation 1 is already calculated, so that the alpha blending process is substantially dispersed.

In this case, the synthesizing unit 306 acquires from the superimposition data acquisition unit 304 an intermediate image for superimposition using the multiplication value αFb ₂ as a pixel value and an α image using the α value as a pixel value. Then, the first term on the right side of Equation 1 is calculated using the α image and the first image, and the pixel value of the display image is obtained by adding the pixel value of the intermediate image. By doing so, the alpha blending process can be made more efficient, and a display image obtained by fusing the first image and the second image can be output with less latency.

Expression 1 is a calculation expression that does not consider the α value of the first image. However, when the transparency is also set for the first image, the brightness F _out of each pixel of the display image can be calculated by the following expression.
α _out = (1−α ₂ ) α ₁ + α ₂
F _out = (α ₁ (1-α ₂ ) Fb ₁ + α ₂ Fb ₂ ) / α _out (Formula 2)

Here, α ₁ and α ₂ are α values respectively set for the pixels of the first image and the second image. When the first image is opaque, that is, when α ₁ = 1, Expression 2 is expressed by Expression 1. As described above, even when an α value is set for each image, intermediate image data having a pixel value of a multiplication value of the α value and the luminance value, for example, α ₁ Fb ₁ and α ₂ Fb _2, is generated. By doing so, the calculation process of Formula 2 performed by the synthesis unit 306 can be made more efficient.

  In these configurations, let us consider a case where the space defining the luminance differs between the first image and the second image. For example, this is the case where the second image is constantly expressed in SDR while the content to be processed is expressed in HDR. In SDR and HDR, even if the data value is the same 10 bits, the luminance value indicated by the data value is different. For this reason, the data cannot be substituted as is in Formula 1 or Formula 2 as equivalent, and luminance conversion is required at any stage before that. In addition to SDR and HDR, if the space defining the brightness is different, it is necessary to align with either space for alpha blending processing.

  In the case where the first image is HDR and the second image is SDR, the brightness of the second image can be expressed in the HDR space so as not to lose the more realistic view of the world expressed in the first image. desirable. However, as described above, when data is acquired in the state of an intermediate image obtained by multiplying the luminance of the image by an α value in order to increase the efficiency of the synthesis process, it is conceivable that the luminance conversion is not correctly performed. In the present embodiment, luminance conversion can be performed accurately with minimum necessary process switching according to the situation. Accordingly, the second image is suitably fused with the first image while allowing a change in the luminance space, and the processing efficiency is not reduced unnecessarily.

  FIG. 4 shows the internal circuit configuration of the information processing apparatus 10. The information processing apparatus 10 includes a CPU (Central Processing Unit) 23, a GPU (Graphics Processing Unit) 24, and a main memory 26. These units are connected to each other via a bus 30. An input / output interface 28 is further connected to the bus 30. The input / output interface 28 includes a communication unit 32 including a peripheral device interface such as USB and IEEE1394, a wired or wireless LAN network interface connected to the network 8, a storage unit 34 such as a hard disk drive or a nonvolatile memory, and a display device. An output unit 36 for outputting data to 16, an input unit 38 for inputting data from the input device 14, and a recording medium driving unit 40 for driving a removable recording medium such as a magnetic disk, an optical disk or a semiconductor memory are connected.

  The CPU 23 controls the entire information processing apparatus 10 by executing an operating system stored in the storage unit 34. The CPU 23 also executes various programs read from the removable recording medium and loaded into the main memory 26 or downloaded via the communication unit 32. The communication unit 32 also establishes communication with an external device such as a server via the network 8 to acquire electronic content data such as a moving image or transmit data generated inside the information processing device 10. Also good.

  The GPU 24 has a function of a geometry engine and a function of a rendering processor, performs a drawing process according to a drawing command from the CPU 23, and stores display image data in a frame buffer (not shown). The display image stored in the frame buffer is converted into a video signal and output to the output unit 36. The main memory 26 is constituted by a RAM (Random Access Memory), and stores programs and data necessary for processing.

  FIG. 5 shows a functional block configuration of the information processing apparatus 10. Each functional block shown in FIG. 5 can be realized in terms of hardware by the configuration of the CPU, GPU, various memories, data bus, etc. shown in FIG. 4, and in terms of software, it is loaded into a memory from a recording medium, It is realized by a program that exhibits various functions such as a data input function, a data holding function, an arithmetic function, an image processing function, and a communication function. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  The information processing apparatus 10 includes an input information acquisition unit 50 that acquires input information from the input device 14, an information processing unit 52 that performs information processing according to a user operation such as a game, and an image generation unit 54 that generates data of a first image. A superimposition data generation unit 56 that generates α value data with the second image, an image synthesis unit 58 that combines the first and second images, and an image output unit 60 that outputs display image data to the display device 16. including.

  The input information acquisition unit 50 is realized by the input unit 38, the CPU 23, and the like, and acquires data indicating the contents of the user operation from the input device 14. Here, the user operation may be performed on a general information processing apparatus, such as selection of an application to be executed or content to be output, start / end of processing, command input, and the like. One of such operations includes a display request for various information represented as a second image.

  When an imaging device or various sensors are introduced as the input device 14, the input information acquisition unit 50 may acquire data such as a captured image and an output value of the sensor. The input information acquisition unit 50 may also acquire electronic content data such as moving images from the server via the network 8. The input information acquisition unit 50 appropriately supplies the acquired data to the information processing unit 52 and the superimposed data generation unit 56 according to the contents.

  The information processing unit 52 is realized by the CPU 23, the main memory 26, and the like, and performs information processing such as a game based on data supplied from the input information acquisition unit 50. As described above, the content of the process performed by the information processing unit 52 is not particularly limited as long as it includes image display. The image generation unit 54 is realized by the GPU 24, the main memory 26, and the like, and generates data of the first image according to a request from the information processing unit 52. For example, when the information processing unit 52 executes a game, the image generation unit 54 draws a game image according to a user operation, a sensor output value, or the like at a predetermined frame rate.

  Alternatively, the image generation unit 54 may decode and expand the moving image data designated by the information processing unit 52. The data of the moving image may be held in the information processing apparatus 10, may be streamed from the server via the network 8, or may be captured on the spot by the imaging apparatus. During a period when it is not necessary to superimpose the second image, the image generation unit 54 outputs the generated first image data to the image output unit 60. During a period in which the second image needs to be superimposed, the image generation unit 54 outputs the generated first image data to the image synthesis unit 58.

  The superimposed data generation unit 56 is realized by the CPU 23, the GPU 24, the main memory 26, and the like, and generates data of the second image as necessary. The second image may be displayed at any time by being called by the user, or may be displayed at a timing determined by the information processing unit 52 in accordance with the progress of the game or video. Alternatively, it may be displayed constantly at one corner of the screen. For the generation of the image, image data held inside may be used, or data acquired by the input information acquisition unit 50 from an external device such as an imaging device or a server may be used.

In addition, the superimposition data generation unit 56 determines, for each pixel, an α value that determines how much the second image is displayed, in association with the second image. The second image may gradually appear or disappear by changing the α value with time. Furthermore, as described above, the superimposition data generation unit 56 in the present embodiment multiplies the luminance value Fb ₂ expressed as the pixel value of the second image by the α value determined for each pixel, thereby obtaining the multiplication value αFb ₂ as a pixel. An intermediate image for superimposition is generated as a value.

When the α value is also set in the first image, the image generation unit 54 sets the α value set in the image, the multiplication value α ₁ Fb ₁ using α ₁ , and the superimposition data generation unit 56 sets the image in the image. Then, an intermediate image is generated in which each of the α value and the multiplication value α ₂ Fb ₂ using α ₂ is a pixel value. However, in the following description, the α value of the second image is simply expressed as “α”. The superimposition data generation unit 56 outputs the generated intermediate image data for superimposition and α image data having an α value as a pixel value to the image synthesis unit 58.

  The image composition unit 58 is realized by the CPU 23, the GPU 24, the main memory 26, and the like. Similar to the image synthesizing apparatus 300 shown in FIG. 3, the image synthesizing unit 58 may be realized as an image synthesizing apparatus that superimposes data of two images and outputs a display image. Specifically, the image composition unit 58 includes a superimposition data acquisition unit 62 that acquires data for superimposition, a first image data acquisition unit 64 that acquires data of the first image, and a luminance space conversion unit that converts a space that defines luminance. 66, a synthesis processing unit 72 that generates display image data by alpha blend processing.

The superimposition data acquisition unit 62 converts superimposition intermediate image data having a pixel value of the multiplication value αFb ₂ of the brightness of the second image and the α value, and α image data having the α value as a pixel value. Obtained from the generation unit 56. The first image data acquisition unit 64 acquires the first image data from the image generation unit 54 and supplies the first image data to the synthesis processing unit 72. The first image data acquisition unit 64 determines whether or not it is necessary to convert the luminance space of the second image based on the luminance space of the first image specified by data analysis or the like.

  Typically, when the second image is steadily expressed in SDR while the first image is expressed in HDR, the first image data acquisition unit 64 needs to convert the luminance space of the second image. Judge that there is. However, if the space for defining the brightness is different, it is determined that conversion is necessary, not limited to the difference in SDR / HDR. Such a situation often occurs when the second image generation function is somewhat fixed on the content execution system side while the first image is created with a high degree of freedom on the content side. Can do.

  When the luminance space needs to be converted, the first image data acquisition unit 64 notifies the superimposed data acquisition unit 62 to that effect. Alternatively, the first image data acquisition unit 64 may notify the luminance space of the first image, and the superimposed data acquisition unit 62 may determine the necessity for conversion in view of both the first image and the second image. The superimposition data acquisition unit 62 switches the supply destination of the acquired data depending on whether or not the luminance space needs to be converted. Specifically, when it is not necessary to convert the luminance space, the superimposed data acquisition unit 62 supplies the acquired data to the synthesis processing unit 72 as it is. When it is necessary to convert the luminance space, the superimposed data acquisition unit 62 supplies the acquired data to the luminance space conversion unit 66.

  When data is acquired directly from the superimposition data acquisition unit 62, the composition processing unit 72 substitutes the data and the first image data acquired from the first image data acquisition unit 64 into the above Equation 1 or Equation 2. To calculate the pixel value of the display image. Since a part of the calculation formula has been calculated, the alpha blend process can be made more efficient and the latency can be suppressed.

The luminance space conversion unit 66 converts the luminance space of the second image using the data supplied from the superimposed data acquisition unit 62. Specifically, the luminance space conversion unit 66 includes a second image data acquisition unit 68 and a data conversion unit 70. The second image data acquisition unit 68 acquires the component of the brightness Fb ₂ of the second image by multiplying each pixel value of the intermediate image for superimposition supplied from the superimposition data acquisition unit 62 by 1 / α. The data converter 70 the luminance Fb ₂ of the second image so obtained is converted to a value of a luminance space of the first image.

  For example, when the luminance range of the first image is larger than the luminance range of the second image, the maximum luminance of the second image is located at 1 / a of the maximum luminance of the first image (where a is a predetermined value satisfying 1 <a). Reduce the entire brightness range. At this time, it is desirable to prevent the image impression from changing before and after conversion by using an appropriate conversion formula such as changing the reduction ratio according to the original luminance.

  Since only the luminance component of the second image is extracted by the second image data acquisition unit 68 and subjected to the conversion process, a suitable result can be obtained even if the formula of spatial conversion from luminance to luminance as conventionally proposed is applied as it is. Can be obtained. The data conversion unit 70 supplies the second image data and the α image thus converted to the composition processing unit 72.

  When the data is acquired from the luminance space conversion unit 66, the composition processing unit 72 displays the data and the first image data acquired from the first image data acquisition unit 64 by substituting the data into the above Equation 1 or Equation 2. The pixel value of the image is calculated. In this case, the composition processing unit 72 first calculates the pixel value of the display image by multiplying the converted luminance of the second image by the α value and appropriately substituting it into the calculation formula. As a result, regardless of the space in which the luminance of the first image is defined, the generation processing of the second image can be suitably fused with the originally intended balance without changing the second image generation processing.

  The composition processing unit 72 outputs the calculated result to the image output unit 60. The image output unit 60 is realized by the GPU 24, the main memory 26, the output unit 36, and the like. The image output unit 60 superimposes the first image data supplied from the image generation unit 54 or the second image supplied from the image composition unit 58. Data is sequentially output to the display device 16 at an appropriate timing.

Next, an effect will be described in which the luminance space conversion unit 66 according to the present embodiment obtains the luminance component of the second image from the intermediate image data for superimposition and then performs the conversion process. FIG. 6 compares the luminance of the display image when the intermediate image is converted as it is and when the luminance component of the second image is acquired and converted according to the present embodiment. In this example, the brightness Fb ₁ of the first image and the brightness Fb ₂ of the second image are both 524 in HDR gradation. In this case, for example, according to the alpha blend calculation of Equation 1, the output value F _out representing the luminance of the display image should be 524 regardless of the α value.

First, the graph 80 is a result of performing the conversion process in a state where the brightness Fb ₂ of the second image is multiplied by the α value. In this case, if the converted result is the second term on the right side of Equation 1, the following equation is obtained as a whole.
F _out = (1−α) Fb ₁ + f (αFb ₂ ) (Formula 3)
Here, f is a function for conversion for expressing the luminance of the second image in the luminance space of the first image. Function f, if (the proviso a predetermined value satisfy 1 <a) 1 / a times it regardless of the original luminance Fb ₂ was a simple linear transformation such as, Equation 3 is replaced by: .
F _out = (1−α) Fb ₁ + αf (Fb ₂ ) (Formula 4)

Expression 4 is nothing but a normal expression for alpha blending with the first image after converting the brightness Fb ₂ of the second image with the function f. On the other hand, when the function f is changed according to the range of the original luminance Fb ₂ or is a predetermined curve such as a power function, the expression 3 is not replaced with a correct expression like the expression 4. Specifically, as illustrated in the graph 80, when α is 0 and 1.0, since F _out = Fb ₁ and f (Fb ₂ ), respectively, an output value F _out = 524 and an accurate value are obtained. As a result, F _out becomes a value different from 524 in the range of 0 <α <1.0.

This is because, in the second term on the right side of Equation 3, the apparent brightness as the input value of the function f becomes smaller than the actual brightness by multiplying the brightness Fb ₂ by an α value of 0 <α <1.0. Because. For example, a coefficient k (Fb ₂ ) that monotonously decreases with an increase in the luminance Fb ₂ so that the smaller the luminance Fb ₂ is, the smaller the degree of reduction in the conversion process is (where 1 / a <k ≦ 1.0, 1 / a is Using the above-described ratio of the maximum brightness in the original space to the maximum brightness of the space when expressed in the space after conversion)
f (Fb ₂ ) = k (Fb ₂ ) · Fb ₂ (Formula 5)
If you give made function, the coefficient k (Fb ₂₎ by ArufaFb ₂ enters the place of Fb ₂ increases.

The effect becomes more remarkable as the α value is smaller. Depending on the setting of k, the brightness Fb ₂ may not be reduced despite the relatively large value, that is, it may be used for the calculation of the alpha blend without being substantially converted. In the example of the graph 80, due to such an effect, the output value _Fout continues to increase until the α value becomes close to 0, and reaches a maximum value of about 720. In such a case, for example, after the second image is displayed, even if the α value is brought close to 0 so that it fades out, the second image becomes rather dark and does not disappear rapidly at a certain point. It will be a natural change.

  On the other hand, according to the present embodiment, since the luminance component of the second image is extracted from the intermediate image for superimposition and the conversion process is performed, Equation 4 is correctly calculated regardless of the α value, as shown in the graph 82. Therefore, an accurate output value 524 is always obtained. That is, the α value setting does not affect the luminance space conversion process, and the intended display effect can be realized. Further, since it is not necessary to introduce a special conversion function in consideration of the α value, an appropriate one can be freely selected from various functions considering only the luminance space. As a result, the degree of freedom in expression and the corresponding range of the luminance space can be further expanded. The same applies when Expression 2 is used for alpha blend calculation.

  Next, the operation of the information processing apparatus realized by the above configuration will be described. FIG. 7 is a flowchart showing a processing procedure in which the information processing apparatus 10 according to the present embodiment generates and outputs a display image. This flowchart is started when a user selects and inputs content such as a game or a video by turning on the information processing apparatus 10. At this time, the information processing unit 52 performs information processing necessary for the selected content, and the image generation unit 54 generates the image as a first image (S10).

  Here, if it is not necessary to display the second image representing various information due to a request from the user, setting as content, or the like (N in S12), the image output unit 60 displays the first image as a display image. By outputting to 16, the image is displayed (S24). When it is necessary to display the second image (Y in S12), the superimposed data generation unit 56 generates data of the second image to be displayed and determines an α value that determines how to display the data for each pixel. Then, an α image having the pixel value as the pixel value and an intermediate image for superimposition obtained by multiplying the second image by the α value are generated (S14).

  The superimposed data generation unit 56 supplies the intermediate image and α image data to the image synthesis unit 58. At this time, the image generation unit 54 also supplies the first image data to the image synthesis unit 58. When conversion of the luminance space of the second image is not necessary, such as when both the first image and the second image are expressed in SDR (N in S16), the composition processing unit 72 of the image composition unit 58 acquires the acquired data. Is used as it is, and an alpha blending process is carried out (S22).

In other words, the display image data is calculated by Expression 1 or Expression 2 using the data obtained from the first image data acquisition unit 64 and the data (αFb ₂ and α) obtained from the superimposed data acquisition unit 62. When conversion of the luminance space is necessary, such as when the first image is expressed in HDR while the second image is SDR (Y in S16), the superimposed data acquisition unit 62 converts the acquired data into the luminance space. By supplying to the conversion unit 66, the luminance space of the second image is converted.

Specifically, first, the second image data acquisition unit 68 acquires the luminance Fb ₂ of the second image by multiplying each pixel value of the intermediate image by 1 / α (S18), and the data conversion unit 70 A space conversion process is performed on the luminance (S20). Then, the composition processing unit 72 performs alpha blend processing using the converted data (S22). That is, the display image data is calculated using the data obtained from the first image data acquisition unit 64 and the data (f (Fb ₂ ) and α) obtained from the luminance space conversion unit 66.

  Regardless of whether or not the luminance space has been converted, the composition processing unit 72 sequentially outputs display image data formed by combining the first image and the second image to the image output unit 60, and the image output unit 60 displays the display device. By outputting to 16, the image is displayed (S24). If it is not necessary to terminate the process due to user operation or content termination (N in S26), the processes in S10 to S24 are repeated for each image frame.

  As a result, the display device 16 can display an additional image combined with the main image of the content during a necessary period regardless of the luminance space represented by the content. If there is no need to update the displayed image frame such as when the second image is a still image, the processes of S14 to S20 may be omitted as appropriate. Each step may be pipelined in a predetermined unit such as a pixel unit. All processes are terminated when the timing for terminating the process is reached by a user operation or the like (Y in S26).

  According to the present embodiment described above, an additional image such as information display is superimposed on the main image of the content by alpha blending, and both images are merged and displayed. For the image to be superimposed at this time, the alpha blending process can be made more efficient by generating an intermediate image for superimposition multiplied by the α value. Further, when the space defining the luminance of the main image is different from the image to be superimposed, a mechanism for returning from the intermediate image to the original image data is provided so that it can be correctly converted into the luminance space matched to the main image.

  As a result, content that does not require conversion can be displayed with low latency and an additional image can be displayed without significantly impairing the world view of the content, while being able to handle content expressed in various luminance spaces. By performing the spatial conversion immediately before the synthesis process, a common mechanism can be used at the image generation stage, and the introduction cost can be kept low. In addition, since only the luminance component of the original image is extracted from the intermediate image and converted, both the conversion process and the alpha blend process can be performed accurately without being influenced by each other. As a result, settings for both processes can be made freely.

  The present invention has been described based on the embodiments. Those skilled in the art will understand that the above-described embodiment is an exemplification, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  For example, in the present embodiment, the case of generating an intermediate image obtained by multiplying the luminance of the second image by the α value has been described. However, the present invention is not limited to this, and may be data that has undergone some calculation on luminance, such as gamma correction. For example, the function of the image composition unit can be similarly applied. That is, when the luminance space needs to be converted, the image composition unit first performs the inverse operation to obtain the luminance data, and then performs the conversion. As in this embodiment, this makes it possible to express the originally intended superimposed image regardless of the luminance space of the main image, and the original processing mechanism can be used as it is in an environment where conversion is not necessary.

  In the present embodiment, an example has been described in which, when the first image is expressed in HDR, the luminance of the second image expressed in SDR is converted in accordance with that, the space before and after the conversion is described. Is not limited to that. For example, when the first image is expressed in SDR and the second image is expressed in HDR, the second image may be converted into an SDR space. A space other than SDR and HDR may be targeted.

  8 network, 10 information processing device, 14 input device, 16 display device, 23 CPU, 24 GPU, 26 main memory, 50 input information acquisition unit, 52 information processing unit, 54 image generation unit, 56 superimposed data generation unit, 58 image A synthesis unit, 60 image output unit, 62 superimposition data acquisition unit, 64 first image data acquisition unit, 66 luminance space conversion unit, 68 second image data acquisition unit, 70 data conversion unit, 72 synthesis processing unit.

Claims (8)

A first image data acquisition unit for acquiring data of the first image;
Each pixel value of the second image generated for superimposed display, the overlay data acquisition unit for acquiring data of the intermediate image for formed by multiplying an alpha value indicating transparency superimposed as part of the alpha blending process,
A second image data acquisition unit for acquiring data of the second image from the intermediate image;
A data conversion unit that converts a luminance value represented by the data of the second image into a value in a luminance space represented by the data of the first image;
A synthesis processing unit that generates and outputs display image data in which the second image is superimposed on the first image by performing alpha blend processing using the converted second image data;
An image composition device comprising:   The first image data acquisition unit identifies a luminance space represented by the acquired first image data, and determines whether or not it is necessary to convert the luminance value represented by the second image data based on the result. The image synthesizing apparatus according to claim 1. The superimposed data acquisition unit supplies the intermediate image data to the synthesis processing unit when it is not necessary to convert the luminance value represented by the data of the second image,
The said synthetic | combination process part implements the remaining alpha blend processes using the data of the said intermediate image, when the data of the said intermediate image are acquired from the said synthetic | combination process part. Image composition device.   The first image data acquisition unit acquires data of the first image expressed in a luminance space defined by electronic content selected by a user, and the superimposed data acquisition unit is commonly defined for electronic content. The image synthesizing apparatus according to any one of claims 1 to 3, wherein data of the intermediate image based on the second image expressed in a luminance space is acquired. An image generation unit that executes electronic content and generates data of the first image;
When necessary to superimpose a second image for superimposing on said first image is generated, after having generated the data of the second image, an intermediate for superimposing multiplied by the alpha value indicating transparency to each pixel value A superimposition data generation unit that performs part of the alpha blending process by generating image data;
An image composition unit that obtains data of the first image and data of the intermediate image, and generates display image data in which the second image is superimposed on the first image;
An image output unit that acquires image data from either the image generation unit or the image synthesis unit and outputs the image data to a display device;
The image composition unit comprises:
A second image data acquisition unit for acquiring data of the second image from the intermediate image;
A data conversion unit that converts a luminance value represented by the data of the second image into a value in a luminance space represented by the data of the first image;
A synthesis processing unit that generates data of the display image by performing an alpha blending process using the data of the second image after conversion;
An information processing apparatus comprising: 6. The composition processing unit according to claim 5, wherein when the luminance value represented by the data of the second image does not need to be converted, the composition processing unit performs the remaining alpha blend processing using the data of the intermediate image. Information processing device. Obtaining data of a first image;
Each pixel value of the second image generated for superimposed display, acquiring data of the intermediate image for formed by multiplying an alpha value indicating transparency superimposed as part of the alpha blending process,
Obtaining data of the second image from the intermediate image;
Converting the luminance value represented by the data of the second image into a value in the luminance space represented by the data of the first image;
Performing alpha blending using the converted second image data to generate display image data in which the second image is superimposed on the first image, and outputting the display image data to a display device; ,
An image composition method by an image composition apparatus characterized by comprising: A function of acquiring data of the first image;
Each pixel value of the second image generated for superimposed display, the function of acquiring the data of the intermediate image for superimposition obtained by multiplying an alpha value indicating transparency as part of the alpha blending process,
A function of acquiring data of the second image from the intermediate image;
A function of converting a luminance value represented by the data of the second image into a value in a luminance space represented by the data of the first image;
A function of generating and outputting display image data in which the second image is superimposed on the first image by performing alpha blend processing using the converted second image data;
A computer program for causing a computer to realize the above.