JP2018136793A - Image processing device, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable update of a background for generating a virtual viewpoint image with high accuracy and a reduced processing load.SOLUTION: An input image divided for each partial region is compared with an input image in a corresponding partial region of an immediately previous frame to derive a ratio of an active region in the partial region. A distance between a position of a center of gravity and a viewpoint position of each partial region is determined from shape data and viewpoint position information. Then, the degree of importance of each partial region is determined from change information and the viewpoint distance information of each partial region. Then, it is determined whether or not to update the input image from the degree of importance for each partial region. When it is determined to make update, the corresponding partial area is updated with the input image, and when it is determined not to make update, the partial area is not updated.SELECTED DRAWING: Figure 1

Description

  The present invention relates to generation of a virtual viewpoint image.

  In recent years, a virtual viewpoint image that reproduces a view from an arbitrarily selected viewpoint (hereinafter referred to as a virtual viewpoint) in a three-dimensional space from images obtained by shooting the same gazing point from different positions with a plurality of cameras. Attention has been paid. When generating a virtual viewpoint image, there are innumerable virtual viewpoints to be selected, and it is impractical to prepare all images corresponding to them. Therefore, conventionally, the reproduction of the appearance from an arbitrary virtual viewpoint is realized by reproducing the photographed subject and the background in a three-dimensional space.

  Here, a method of reproducing the background in the three-dimensional space is realized by pasting a plurality of background images taken on shape data (background model) representing a background prepared in advance to corresponding locations. When a moving image is input, a process of updating an image to be pasted on the background model is performed every time a background image is input.

  In addition, due to the recent increase in resolution of captured images, the data transmission bandwidth is compressed and the processing load increases. To generate a virtual viewpoint image from a plurality of captured images in real time, a background image that occupies a high proportion of the captured images. Must be processed efficiently.

  As a technique for efficiently processing an image, Patent Document 1 describes a processing load by dividing a computer screen into partial areas and compressing a partial area in which temporal changes occur frequently to reduce the amount of data. A technique for reducing the above is disclosed.

JP 2011-238014 A

  However, even if the amount of data in a frequently changing area is reduced as in the technique described in Patent Document 1, a virtual viewpoint image using a natural image frequently changes in many areas. For this reason, the data amount reduction effect is small, and the reduction of the processing load is not sufficient. In addition, image processing in an insignificant area such as an area away from the virtual viewpoint is not sufficiently suppressed, and the processing load is not sufficiently reduced.

  In view of the above-described problems, an object of the present invention is to update a background for generating a virtual viewpoint image with high accuracy and low processing load.

The image processing apparatus according to the present invention has the following configuration, for example. That is, the acquisition unit that acquires change information regarding temporal changes for each of the plurality of partial areas in the input image, the determination unit that determines the attention information about the degree of attention for each of the plurality of partial areas, and the acquisition unit. Determining means for determining the importance for each of the plurality of partial areas from the change information and the attention information determined by the determining means, and updating the partial area according to the importance determined by the determining means Updating means.

  According to the present invention, the background for generating the virtual viewpoint image can be updated with high accuracy and with a low processing load.

It is a block diagram which shows the function structural example of the image processing apparatus in 1st Embodiment. It is a block diagram which shows the hardware structural example of the image processing apparatus which concerns on embodiment. It is a figure which shows the example of the shape data showing the rectangular parallelepiped shape of the some triangular mesh. It is a figure which shows the example of the input image containing the gazing point image | photographed from three cameras. It is a flowchart which shows an example of the process sequence which updates the background image which concerns on 1st Embodiment. It is a figure which shows the example which divided | segmented the input image into the partial area | region by shape data. It is a figure which shows the example which acquired the change information of each partial area | region in shape data. It is a figure which shows the example which determined the distance of the gravity center position of each partial area | region and the gaze point. It is a figure which shows the example of the determination conditions of the importance in 1st Embodiment. It is a figure which shows the example which calculated | required the importance of each partial area | region in 1st Embodiment. It is a block diagram which shows the function structural example of the image processing apparatus in 2nd Embodiment. It is a flowchart which shows an example of the process sequence which updates the background image which concerns on 2nd Embodiment. It is a figure for demonstrating the field angle of a virtual viewpoint, and the parameter | index for every partial area. It is a figure which shows the example of the determination conditions of the importance in 2nd Embodiment. It is a figure which shows the example which calculated | required the importance of each partial area | region in 2nd Embodiment. It is a block diagram which shows the function structural example of the image processing apparatus in 3rd Embodiment. It is a flowchart which shows an example of the process sequence which updates the background image which concerns on 3rd Embodiment. It is a figure which shows the background containing an attention object. It is a figure which shows the example of the determination result of the distance with an attention object. It is a figure which shows the example which calculated | required the importance of each partial area | region in 3rd Embodiment.

(First embodiment)
Hereinafter, a first embodiment for carrying out the present invention will be described in detail with reference to the drawings. In the present embodiment, in order to reproduce the background in the three-dimensional space as a pre-stage for generating the virtual viewpoint image, the image is updated for each partial region of the background shape to correspond to the background change in the three-dimensional space. . Specifically, paying attention to the distance from the position of the gaze point related to the generation of the virtual viewpoint image, the importance is determined based on the distance from the position of the gaze point and the temporal change, and the image is selected according to the degree of importance. Update process. As a result, image processing in an area not noticed by the user is suppressed, and the processing load is reduced.
Note that the virtual viewpoint image in the present embodiment is an image generated by simulating an image obtained when a subject is imaged from a virtual viewpoint. In other words, it can be said that the virtual viewpoint image is an image representing the appearance at the virtual viewpoint. The virtual viewpoint (virtual viewpoint) may be designated by the user, or may be automatically designated based on the result of image analysis or the like. Free viewpoint images, arbitrary viewpoint images, and the like exist as concept words similar to virtual viewpoint images. The system of the present embodiment is not limited to the virtual viewpoint image, and can similarly process a free viewpoint image and an arbitrary viewpoint image.

FIG. 2 is a block diagram illustrating a hardware configuration example of the image processing apparatus 10 according to the present embodiment.
In FIG. 2, the CPU 201 controls the entire image processing apparatus 10 and performs various processes according to programs stored in the ROM 202. The ROM 202 stores the above-described program and other information. When the above-described program is executed, the RAM 203 is expanded from the ROM 202 and used as a work area for the CPU 201.

  The input device 204 includes an operation member for inputting a user instruction. A communication I / F 205 is an interface for transmitting / receiving data to / from an external device via a network or the like. The external I / F 206 is an interface for outputting data stored in the storage device 207 to an external device such as a printer or a display device. The storage device 207 is, for example, a memory for storing image data, and stores data input from the communication I / F 205 or the like, or data processed by the CPU 201.

  Next, a functional configuration of the image processing apparatus 10 according to the present embodiment will be described. In the present embodiment, the image processing apparatus 10 inputs image data captured by three cameras having the same gazing point as the center of the angle of view.

FIG. 1 is a block diagram illustrating a functional configuration example of an image processing apparatus 10 according to the present embodiment. In the present embodiment, each function shown in FIG. 1 is realized by the CPU 201 in FIG. 2 executing a predetermined program to control each unit. However, some or all of the functions shown in FIG. 1 are executed based on processing by one or more processors such as ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), etc. You may be made to do.
In FIG. 1, the input unit 101 inputs background shape data, gazing point position information, a plurality of image data, and camera information, and stores these pieces of information in the storage device 207. Here, the background shape data is a set of triangular or quadrilateral meshes, and indicates the shape of the background to be imaged, such as a stadium or a concert hall. The function of the input unit 101 is realized by using at least one of the communication I / F 205 and the input device 204 in FIG.

  FIG. 3 shows shape data 300 representing a rectangular parallelepiped shape without a ceiling surface as a set of a plurality of triangular meshes 301 as an example of shape data. The triangular mesh is represented by three-dimensional coordinates of a triangular point, and the coordinate system is expressed with the center position 302 of the bottom surface as the origin. Hereinafter, a rectangular parallelepiped mesh model shown in FIG. 3 will be described as an example. In the present embodiment, the shape data is represented by a set of meshes, but is not limited to this. Further, even if the shape data is represented by a set of meshes, it is not limited to a triangular mesh. That is, the shape data 300 can be represented by a set of arbitrary shapes.

The gazing point position information is the three-dimensional coordinates of the center position of the imaging range in a plurality of image data input for generating a virtual viewpoint image. The point-of-gaze position information is expressed in the same coordinate system as the shape data 300 shown in FIG. , Z) = (50, 0, 0).
The plurality of image data are image data (hereinafter referred to as input images) respectively captured by the three cameras described above.

FIG. 4 is a diagram illustrating an example of an input image including a gazing point photographed from three cameras. FIG. 4A is a diagram for explaining a state in which the same gazing point 401 is imaged at the center of the angle of view in three different directions 402 to 403 from three cameras arranged at different positions. FIGS. 4B to 4D show examples of input images taken by the respective cameras, and the point of gaze 401 is located at the center of each image. The positions of the gazing points in the three-dimensional space in FIGS. 4B to 4D are common.
The camera information is information indicating the positions, postures, and setting values of a plurality of cameras that have captured the above-described input images.

  Based on the camera information, the image selection unit 102 specifies a corresponding region between the input image and the shape data, and divides the input image into a plurality of partial regions indicated by the shape data. Then, an input image to be used as an image processing target is selected for each divided partial area, and the shape data and the divided input image are output to the change information acquisition unit 103 and the attention level determination unit 104. The division into partial areas and the selection method of the input image will be described later.

The change information acquisition unit 103 compares the input image divided for each partial area with the input image of the corresponding partial area of the immediately preceding frame, acquires the ratio of the area that has changed in the partial area, The information is output to the importance determining unit 105 as area change information.
The attention level determination unit 104 determines the distance between the centroid position and the gazing point position of each partial region from the shape data and the gazing point position information, and outputs it to the importance level determination unit 105 as gazing point distance information (attention information). To do.

The importance level determination unit 105 determines the importance level of each partial area from the change information of each partial area acquired by the change information acquisition unit 103 and the gaze point distance information calculated by the attention level determination unit 104. The determination method will be described later.
The image update unit 106 determines whether or not to update the input image from the importance for each partial region. If it is determined to be updated, the corresponding partial area is updated with the input image. If it is determined not to be updated, the partial area is not updated. The update method will be described later.
The image output unit 107 outputs the image data in which the partial area has been updated to an external device that generates a virtual viewpoint image. The function of the image output unit 107 is realized by the communication I / F 205 and the CPU 201 in FIG.

FIG. 5 is a flowchart illustrating an example of a processing procedure for updating a background image by the image processing apparatus 10 according to the present embodiment.
In step S501, the image selection unit 102 divides the input image into partial areas of the shape data 300 represented by a plurality of triangular partial areas illustrated in FIG. 3 based on the camera information. In this embodiment, an example in which the entire input image is divided into partial areas will be mainly described. However, the entire input image may not necessarily be divided into partial areas. Using the camera information indicating the position and orientation of the camera that captured the input image, it is possible to determine the correspondence of which position in the shape data each captured image was captured, and the triangle of the input image represented by the shape data It can be divided into partial areas. FIGS. 6A to 6C show examples in which the input images shown in FIGS. 4B to 4D are divided into partial areas using the shape data shown in FIG. . In FIG. 6, broken lines indicate the boundaries of the partial areas, and each partial area is represented by a triangle according to the shape data. However, as described above, the shape data does not necessarily have to be represented by a set of triangular partial areas, and can be represented by a set of arbitrary shapes.

  In step S <b> 502, the image selection unit 102 selects an input image to be pasted on the shape data for each partial area. Depending on the partial area, a plurality of input images may exist overlappingly, and it is necessary to select an image to be used as an image of the partial area. As a selection method, an input image having the largest ratio including the corresponding partial region is selected. However, the method is not limited to the method of selecting the largest input image, and the input image may be selected according to a predetermined selection policy. For example, an input image including a predetermined position of the partial area may be selected, or an input image including the largest number of vertices of the partial area may be selected. When the selection of the input image is completed for all the partial areas, the process proceeds to the next S503.

  Hereinafter, an input image selection method will be described with reference to FIG. For example, the partial area A existing in the vicinity of the gazing point 401 is included in the three input images shown in FIGS. 6 (a) to 6 (c). In the input images shown in FIG. 6A and FIG. 6B, a part of the partial area A protrudes from the input image. However, in the input image shown in FIG. Existing. Therefore, in the case of the partial area A, the input image of FIG. 6C having the largest ratio of the partial area A is selected as the image of the partial area. In this embodiment, the input image having the maximum ratio including the partial area is selected. However, the input image is not limited to this, and the input image is selected based on other criteria such as selecting the input image having the maximum resolution of the partial area. Also good.

  Next, the processing of S503 to S506 described below is repeated for each partial region. First, in step S503, the change information acquisition unit 103 generates change information based on a ratio (change value) of the number of pixels having a difference from the immediately preceding frame in the target partial region. FIG. 7 is a diagram illustrating an example of a change value of each partial region in the shape data of FIG. In FIG. 7, the shape data of FIG. 3 is shown expanded, and the number written in each partial area indicates the ratio (%) of the area having a difference from the immediately preceding frame. In the example of FIG. 7, it is shown that the change at the side surface portion of the rectangular parallelepiped is large and the change at the bottom surface portion is small. In the present embodiment, the change value (change information) is the ratio of the number of pixels having a difference from the immediately preceding frame, but the average value of the difference value from the immediately preceding frame and the difference from the immediately preceding frame have continued for a certain period. The number of pixels may be used as a change value. In the present embodiment, the example in which the change information acquisition unit 103 of the image processing apparatus 10 calculates change information has been mainly described. However, for example, a configuration in which change information is acquired from another apparatus may be employed. good.

  In step S <b> 504, the attention level determination unit 104 calculates the distance between the gravity center position of the target partial region and the gazing point, and generates distance information. The distance is determined from gazing point position information and shape data including coordinate information of each partial region. FIG. 8 shows an example in which the distance between the center of gravity of each partial area and the gazing point 401 in the shape data of FIG. 3 is determined, and the value written in each partial area indicates the distance from the gazing point. In this embodiment, the barycentric position is used as the distance between the gazing point and the partial area. However, the present invention is not limited to this, and the distance from the position in the partial area closest to the gazing point may be used. Further, in the present embodiment, an example in which the attention level determination unit 104 of the image processing apparatus 10 calculates the distance has been mainly described. However, for example, a configuration in which information about the distance is acquired from another apparatus may be employed. good.

  Next, in S505, the importance level determination unit 105 determines the importance level of the target partial region from the change information generated in S503 and the distance information generated in S504. Hereinafter, the criteria for determining the importance will be described.

  FIG. 9 is a diagram illustrating an example of the importance determination conditions. In the case of the conditions shown in FIG. 9, if the change value is less than 10%, the change is small compared to the image updated immediately before. In addition, although the change value is 10% or more, even when the distance from the gazing point is 120 or more, it is considered that the distance from the gazing point is far away and the user rarely pays attention to the area as a virtual viewpoint, The importance is “low”.

  On the other hand, when the change value is 10% or more and the distance from the gazing point is 80 or more and less than 120, there is a change from the image updated last time, and the user may pay attention not to be far from the gazing point. It is considered that there is a certain importance level. Furthermore, when the change value is 10% or more and the distance from the gazing point is less than 80, it is considered that there is a change from the previously updated image and the user is likely to pay attention near the gazing point. " In this embodiment, the example in which the importance level determination unit 105 of the image processing apparatus 10 determines the importance level is mainly described. However, for example, a configuration in which information on the importance level is acquired from another apparatus is adopted. May be.

  FIG. 10 is a diagram showing an example in which the importance of each partial region is obtained from the conditions shown in FIG. The degree of importance is set higher for a partial region that changes greatly from the immediately preceding frame and is closer to the point of interest. As described above, the importance of each partial region is determined based on the change value and the distance from the gazing point. In the present embodiment, the change information and the distance from the gazing point are determined by threshold values, and the importance is obtained in three stages. However, the present invention is not limited to this, and may be expressed by a numerical value or the like.

  Next, in step S506, the image update unit 106 determines whether to update the image according to the importance determined in step S505, and performs processing for each partial region. Here, in the case of a partial area with a high importance level, the image update unit 106 updates the partial area with the input image, and stores the image related to the partial area in the storage device 207. In the partial area with the importance level of “medium”, the update frequency is thinned out by half, and in the case of updating, the image update unit 106 updates using the image obtained by reducing the resolution of the input image by 30%. Is stored in the storage device 207. In the partial area with the importance level “low”, when updating the update frequency at a time per second, the image update unit 106 updates the image using an image obtained by reducing the resolution of the input image by 50%. The image relating to the area is stored in the storage device 207. By performing the processing according to the importance in this way, the update frequency and the data amount of the low importance area are reduced, and the average processing load of the image processing apparatus 10 is reduced.

  When the partial area is updated, the image output unit 107 outputs an image related to the partial area to the external device. The external device can update the background in the three-dimensional space by pasting the image related to the updated partial region to the background model, and generate a virtual viewpoint image from the background in the updated three-dimensional space. In this way, the processing load of transmission by the image processing apparatus 10 can also be reduced.

  In the present embodiment, the update frequency and resolution of the input image are changed according to the importance. However, the present invention is not limited to this, and the color depth of the image to be updated is reduced or the image to be updated is compressed. You may go. As described above, according to the present embodiment, the importance is determined from the distance from the position of the point of interest of each partial region and the change value of the image, and the image update frequency and resolution are controlled according to the importance. Thereby, in the external device, it is possible to generate a high-quality image in an area that the user pays attention to and reduce the processing load.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described. In the present embodiment, as an index as to whether or not the region is a region of interest to the user, whether or not it is included in the angle of view of the virtual viewpoint that is an arbitrary viewpoint in the three-dimensional space selected by the user is used as an index. Then, the change information, the index value based on the virtual viewpoint, and the load information are used to determine the importance in order to suppress the image processing when the image processing apparatus is overloaded. The hardware configuration of the image processing apparatus according to this embodiment is the same as that shown in FIG.

FIG. 11 is a block diagram illustrating a functional configuration example of the image processing apparatus 1100 according to the present embodiment. Only differences from FIG. 1 will be described below.
The input unit 101 inputs background shape data, virtual viewpoint information, a plurality of image data, and camera information, and stores these pieces of information in the storage device 207. Here, the virtual viewpoint information is information indicating the position and orientation of the virtual viewpoint, similar to the camera information, and is input from the input device 204 by a user operation, for example. The attention level determination unit 104 specifies a partial region included in the angle of view of the virtual viewpoint based on the input virtual viewpoint information, and outputs the partial area to the importance level determination unit 105.

  The processing load management unit 1101 monitors the processing load state of the image processing apparatus 1100, and outputs a value indicating the load state to the importance level determination unit 105. The value indicating the state of the processing load corresponds to the usage rate of the CPU 201 in the image processing apparatus 1100, for example. In this embodiment, the usage rate of the CPU 201 is used as information indicating the state of the processing load. However, the present invention is not limited to this. For example, the usage rate of the RAM 203 may be used, or the usage rate of the CPU 201 and the usage of the RAM 203 may be used. A combination of rates may be used.

FIG. 12 is a flowchart illustrating an example of a processing procedure for updating a background image by the image processing apparatus 1100 in the present embodiment. Note that description of the same processing as in the first embodiment is omitted.
In S1201, the attention level determination unit 104 determines whether or not the corresponding partial area is included in the angle of view of the virtual viewpoint, and determines an index of the partial area. Based on the virtual viewpoint information, a partial area included in the virtual viewpoint angle of view is specified by projecting the angle of view of the virtual viewpoint onto shape data.

  FIG. 13 is a diagram for describing processing for determining an index by determining whether or not an angle of view is included in a virtual viewpoint. As illustrated in FIG. 13A, when virtual viewpoint information for designating a virtual viewpoint in the direction indicated by the arrow 1301 is input from the input device 204 by the user operation, the image illustrated in FIG. A corner is assumed. Then, when applied to the shape data 300 of FIG. 3, there are three partial areas in the angle of view of the virtual viewpoint shown in FIG. Also, the index of the partial area included in the virtual viewpoint is 2, the partial area in contact with the partial area included in the virtual viewpoint is 1, and the index of the other partial areas is 0. In this case, the index of each partial area has a distribution as shown in FIG. It should be noted that the adjacent area of the partial area included in the virtual viewpoint has an index of 1 in consideration of errors in the angle of view determination of the virtual viewpoint.

  In step S <b> 1202, the processing load management unit 1101 detects a CPU usage rate that is a value indicating the processing load of the image processing apparatus 1100. In step S <b> 1203, the importance level determination unit 105 determines the importance level based on the change information of the corresponding partial area, the index based on the virtual viewpoint, and the CPU usage rate.

  FIG. 14 is a diagram illustrating an example of conditions for determining importance based on change information and an index based on a virtual viewpoint when the CPU usage rate is low (less than a predetermined value). For example, when the change information of each partial area is as shown in FIG. 7 and the index from the virtual viewpoint of each partial area is as shown in FIG. 13C, the importance of each partial area is as shown in FIG. The distribution is as shown in (a). Further, when the CPU usage rate of the image processing apparatus 1100 is high (above a predetermined value), the importance is lowered by one step in order to reduce the overall processing load. That is, the overall processing load of the image processing is reduced as the importance shown in FIG. 15B, and an overload state of the image processing apparatus 1100 is avoided.

  As described above, according to the present embodiment, the importance is determined from the angle of view of the virtual viewpoint input by the user, and the image update frequency and resolution are controlled according to the importance. As a result, it is possible to generate a high-quality image in the region where the user is viewing and to reduce the processing load. In this embodiment, the case where there is a single virtual viewpoint is shown. However, when a plurality of virtual viewpoints are input, whether or not each partial area is included in the angle of view of the virtual viewpoint is determined for each virtual viewpoint. judge. Then, the number of virtual viewpoints whose partial angles are included in the angle of view may be used as an index value of the attention level of each partial area. For example, when there are three virtual viewpoints and the target partial area is included in the angle of view of two of the three virtual viewpoints, the index of the partial area is set to 2.

(Third embodiment)
Hereinafter, a third embodiment of the present invention will be described. In the present embodiment, as an indicator of whether or not the region is a region of interest to the user, attention is paid to the distance from the subject to which the user is interested, and the change information, the distance to the subject to which the user is interested, and the image output are obtained. The degree of importance is determined for each partial area using the required performance, which is the processing time to be obtained. The hardware configuration of the image processing apparatus according to this embodiment is the same as that shown in FIG.

FIG. 16 is a block diagram illustrating a functional configuration example of the image processing apparatus 1600 according to the present embodiment. Only differences from FIG. 1 will be described below.
The input unit 101 inputs background shape data, target subject information, a plurality of image data, and camera information, and stores these pieces of information in the storage device 207. Here, the subject of interest information is information indicating the position coordinates of the subject of interest of the user in the three-dimensional space, and is input from the input device 204 by the user's operation, for example. The attention level determination unit 104 determines the distance between the subject related to the target subject information and each partial area, and outputs the distance to the importance level determination unit 105.

  The required performance management unit 1601 manages the time until output from the image output unit 107 in response to a user request as a required performance value, and sends information on the required performance value to the importance level determination unit 105. For example, when the user requests real-time processing, it is necessary to output at the frame rate of the input image, and in the case of 60 fps (Frame Per Second), 1000 ms / 60 = 16 ms is managed as the required performance value.

FIG. 17 is a flowchart illustrating an example of a processing procedure for updating a background image by the image processing apparatus 1600 in the present embodiment. Note that description of the same processing as in the first embodiment is omitted.
In step S <b> 1701, the attention level determination unit 104 specifies the distance between the position of the subject of interest and each partial area from the subject of interest information and the shape data. The reference point when measuring the distance to each partial area is the center of gravity of each partial area, as in the first embodiment.

  In step S1702, the required performance management unit 1601 detects the current required performance value. In step S <b> 1703, the importance level determination unit 105 determines the importance level based on the distance between the corresponding partial region and the subject of interest, the change information, and the required performance value.

  For example, when the subject 1801 shown in FIG. 18 is designated as the subject of interest, the distance to each partial region is determined as shown in FIG. In addition, FIG. 20 shows the result of importance determined by the same criteria as in the first embodiment. 20A shows a case where the required performance is below the threshold, and FIG. 20B shows a case where the required performance is above the threshold. When the required performance is large, the processing load is reduced by reducing the importance.

  As described above, according to the present embodiment, the importance is determined from the distance of the object of interest, and the image update frequency and the resolution are controlled according to the importance. As a result, it is possible to generate a high-quality image in a region that the user pays attention to and reduce the processing load.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 103 Change information acquisition part 104 Attention degree determination part 105 Importance determination part 106 Image update part

Claims (13)

Acquisition means for acquiring change information regarding temporal changes for each of a plurality of partial areas in the input image;
Determining means for determining attention information regarding the degree of attention for each of the plurality of partial areas;
Determining means for determining importance for each of the plurality of partial areas from the change information acquired by the acquiring means and the attention information determined by the determining means;
Updating means for updating the partial area according to the importance determined by the determining means;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the updating unit updates each partial region at an update frequency according to the importance.   The image processing apparatus according to claim 1, wherein the updating unit updates each partial area with a resolution corresponding to the importance level.   The image processing apparatus according to claim 1, wherein the determination unit determines, as the attention information, a distance from a gazing point position of the input image for each partial region.   The image processing according to claim 1, wherein the determination unit determines, for each partial area, an index as to whether or not the attention information is included in an angle of view of a virtual viewpoint. apparatus.   The image processing apparatus according to claim 5, wherein the determination unit determines, for each partial area, the number of virtual viewpoints including a partial area in an angle of view as the attention information.   The image processing apparatus according to claim 5, wherein the determination unit determines the importance according to a processing load of the image processing apparatus.   The image processing apparatus according to claim 1, wherein the determination unit determines a distance from an object of interest as the attention information for each partial region.   The image processing apparatus according to claim 8, wherein the determination unit inputs position information of the object of interest and determines a distance from the object of interest.   The image processing apparatus according to claim 8, wherein the determination unit determines the importance level according to the required performance of the image processing apparatus.   The image processing apparatus according to claim 1, further comprising an output unit that outputs an image related to the partial area updated by the update unit to an external device. An acquisition step of acquiring change information regarding a temporal change for each of a plurality of partial areas in the input image;
A determination step of determining attention information regarding the degree of attention for each of the plurality of partial regions;
A determination step of determining the importance for each of the plurality of partial regions from the change information acquired in the acquisition step and the attention information determined in the determination step;
An update step of updating the partial area according to the importance determined in the determination step;
An image processing method comprising: An acquisition step of acquiring change information regarding a temporal change for each of a plurality of partial areas in the input image;
A determination step of determining attention information regarding the degree of attention for each of the plurality of partial regions;
A determination step for determining the importance for each partial region from the change information acquired in the acquisition step and the attention information determined in the determination step;
An update step of updating the partial area according to the importance determined in the determination step;
A program that causes a computer to execute.

Images