JP7455546B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program.

Conventionally, in order to draw an object in a three-dimensional space, texture mapping has been performed in which texture data representing the color of the object's surface is mapped to data representing the three-dimensional shape of the object. For texture mapping, a texture map is used in which texture data is expanded onto a two-dimensional plane and saved in an image format.

Patent Document 1 selects a viewpoint from which the area of a triangle projected from a polygon mesh of triangles that constitutes a polygon mesh model representing a three-dimensional shape of an object is the largest, and generates a texture map using texture data from that viewpoint. Discloses that. In this technique, a triangle to which texture data is assigned is transformed into a right-angled isosceles triangle, and these triangles are laid out on a texture map.

Japanese Patent Application Publication No. 2004-227095

However, in the method of Patent Document 1, the triangle to which texture data is assigned is transformed into a right isosceles triangle. Therefore, if texture mapping is performed using a texture map based on the texture data to draw an object, there is a possibility that the image quality of the object will deteriorate.

The present invention has been made in view of such problems, and an object of the present invention is to generate a texture map so that the image quality of a drawn object does not deteriorate.

In one embodiment of the present invention, an image processing device that generates a texture map that corresponds to three-dimensional shape data of an object and represents at least a color of the object, the One viewpoint is selected from among a plurality of viewpoints for an object, and a plurality of components for which the same viewpoint is selected and that are directly and indirectly adjacent to the component are grouped together, and the plurality of components are combined into one group. a grouping means for classifying into groups; a setting means for setting a mapping obtained by projecting a plurality of components belonging to the group onto the selected viewpoint on a two-dimensional map as an area corresponding to the group; The image forming apparatus includes determining means for determining pixel values corresponding to the region from the viewpoint image, and generating means for generating a texture map based on the determined pixel values corresponding to the region.

According to the present invention, a texture map can be generated without deteriorating the image quality of a drawn object.

FIG. 1 is a functional block diagram of an image processing device in a first embodiment. It is a flowchart of image processing in the first embodiment. It is a flowchart of grouping processing in a first embodiment. It is a fragment arrangement processing flowchart in a first embodiment. FIG. 3 is a diagram showing objects in a three-dimensional space and imaging viewpoints in the first embodiment. FIG. 3 is a diagram showing a captured image and a fragment of a polygon island in the first embodiment. FIG. 3 is a diagram showing textured polygon mesh data in the first embodiment. FIG. 1 is a hardware configuration diagram of an image processing device in a first embodiment. It is a flowchart of fragment arrangement processing in a second embodiment. FIG. 3 is a diagram schematically explaining fragment placement processing in the first embodiment. FIG. 7 is a diagram schematically illustrating fragment placement processing in the second embodiment.

Embodiments of the present invention will be described in detail below with reference to the drawings. Note that the following embodiments do not limit the present invention. Furthermore, not all combinations of features described in the embodiments below are essential to the present invention. Note that the same configurations will be described using the same reference numerals.

<First embodiment>
In the first embodiment, images captured by a plurality of imaging devices arranged to surround an object in a three-dimensional space and a polygon mesh representing the three-dimensional shape of the object are input, and textured polygons are Explain how to generate a mesh model. A polygon mesh model is data that expresses the three-dimensional shape of an object (ie, three-dimensional shape data). The texture of a polygon mesh is expressed by describing pixel values (color information) of the texture on a texture map. A texture map in which pixel values are described and saved in an image format is also called a texture image. Note that it is assumed that the positional relationship between the plurality of imaging devices has been calibrated, and that the respective calibration parameters are stored in advance.

FIG. 1 is a functional block diagram of an image processing apparatus in this embodiment. The image processing apparatus in this embodiment includes a viewpoint setting section 100, a model acquisition section 101, a grouping section 102, a fragment forming section 103, a placement section 104, and a texture value determining section 105.

The viewpoint setting unit 100 acquires the calibration parameters of the viewpoint of each imaging device from a storage unit (not shown).

The model acquisition unit 101 acquires polygon mesh data representing the three-dimensional shape of an object from a storage unit (not shown).

The grouping unit 102 classifies a plurality of polygons forming a polygon mesh into a plurality of groups using the acquired calibration parameters, and generates an index list of polygons belonging to each group. In this embodiment, a group of polygons is called a polygon island.

The fragment forming unit 103 forms, for each polygon island, a fragment (also called a fragment) that defines the size and shape of the polygon island on the texture map.

The placement unit 104 places each fragment formed for each polygon island on a texture map, calculates texture coordinates (two-dimensional positions on the texture map) corresponding to the vertices of the polygon, and creates a polygon mesh containing texture coordinate information. The data is stored in a storage unit (not shown).

The texture value determining unit 105 acquires captured images corresponding to the viewpoints of each imaging device from a storage unit (not shown), writes color information representing the surface color of each polygon island on a texture map, and stores the image in a storage unit (not shown). Save it as image data.

In this way, the textured polygon mesh model in this embodiment is described by polygon mesh data including texture coordinate information and image data of a texture map.

Next, the flow of image processing in this embodiment will be explained using FIG. 2. The series of processes shown in the flowchart of FIG. 2 is performed by the CPU of the image processing device loading a control program stored in the ROM or HDD into the RAM and executing it. Alternatively, some or all of the functions of the steps in the flowchart may be realized by hardware such as an ASIC or an electronic circuit. The symbol "S" in the description of the flowchart means a "step" in the flowchart. The same applies to other flowcharts.

First, in S200, the viewpoint setting unit 100 sets viewpoint calibration parameters for each imaging device.

In S201, the model acquisition unit 101 acquires polygon mesh data of the object.

In S202, the grouping unit 102 selects, for each polygon, the viewpoint that images the polygon with the highest image quality, and integrates adjacent polygons from which the same viewpoint is selected into one group. Details will be described later with reference to FIG. The grouping unit 102 outputs an index list of polygons (that is, polygon islands) for each group.

In S203, the fragment forming unit 103 projects the polygons belonging to each polygon island onto the selected viewpoint, and forms a two-dimensional fragment of each polygon island that is the mapping. As will be described later, the color information of the texture of each polygon island is described according to the shape and size of the formed fragment. The fragment information holds a circumscribed rectangle of the mapping (specifically, the upper left pixel position, width, and height of the circumscribed rectangle) and a fragment map (a map of pixels that overlap with the mapping within the circumscribed rectangle).

In S204, the placement unit 104 places all the fragments on one texture map, and makes the coordinates of the upper left of the circumscribed rectangle of the fragment to be placed correspond to the vertices of the polygon included in the polygon island paired with the fragment. Calculate texture coordinates on the texture map. That is, the arrangement unit 104 sets all the fragments on the two-dimensional map. Details will be described later with reference to FIG.

In S205, the texture value determination unit 105 reads the captured image and copies the pixel values of pixels that overlap the fragments at the selected viewpoint onto the texture map, thereby determining the pixel values of the texels on the texture map corresponding to each fragment. decide. Note that the method for determining the pixel values (for example, color information) of the texture map is not limited to this, and may include mixing the pixel values of captured images from viewpoints other than the selected viewpoint, or using the pixel values of the viewpoint selected by occlusion determination. Other methods, such as not being used, may also be used.

Next, the above processing flow will be schematically explained using FIGS. 5 to 7.

FIG. 5 shows a polygon mesh 503 representing the shape of a cubic object in three-dimensional space, and viewpoints 500 to 502 of three imaging devices arranged to surround it. Here, for the sake of simplicity, a process for generating textures for polygons F0 to F3 among a plurality of polygons forming the polygon mesh 503 will be described. Polygons F0 to F3 are composed of vertices V0 to V5.

FIG. 6 shows images of the polygon mesh 503 captured from the three viewpoints described above, and fragments of the polygon island projected onto the two-dimensional space of each image. Images 600-602 show images taken from each of three viewpoints 500-502. In this example, polygon F0 and polygon F1 are classified into the same group and belong to the same polygon island, and form a fragment 610 projected onto the two-dimensional space 603 of the image captured from the viewpoint 500. Furthermore, polygons F2 and F3 are classified into the same group and belong to the same polygon island, and form a fragment 611 projected onto the two-dimensional space 605 of the image captured from the viewpoint 502. Fragment 610 is represented by a region surrounded by vertices C0 to C3. Vertices C0 to C3 indicate projection positions (texture coordinates) corresponding to vertices V0 to V4 of polygon mesh 503, respectively. Further, the fragment 611 is represented by an area surrounded by vertices C4 to C7. Vertices C4 to C7 indicate projection positions corresponding to vertices V1 to V5 of polygon mesh 503, respectively. Note that since the image captured from the viewpoint 501 was not selected for the polygons F0 to F3, no fragment is formed in the two-dimensional space 604 of the image.

FIG. 7 shows an example of textured polygon mesh data generated by arranging the above-described fragments on a texture map. FIG. 7A shows a table that describes the vertex coordinates of vertices V0 to V5 of the polygon mesh 503. FIG. 7B shows a table that describes the texture coordinates of vertices C0 to C7 of fragments 610 and 611. FIG. 7C shows a table that describes polygon information of the polygon mesh 503, and shows the correspondence between vertices V0 to V5 of polygons F0 to F3 constituting the polygon mesh 503 and vertices C0 to C7 of fragments 610 and 611. is described. FIG. 7(D) shows a texture map described as image data.

Next, a detailed process flow of the grouping process (S202) of polygons forming a polygon mesh will be described with reference to the flowchart of FIG.

First, in S300, the grouping unit 102 selects the viewpoint with the highest resolution for each polygon that makes up the polygon mesh (that is, for each component that makes up the three-dimensional shape of the object). Specifically, among the viewpoints where the mapping of each polygon is within the angle of view and the surface is visible (that is, the viewpoint where the inner product of the surface normal and the direction vector from the polygon to the viewpoint position is positive), Select the viewpoint with the largest area. However, the method of selecting the viewpoint is not limited to this, for example, selecting the viewpoint that is most directly facing the surface, considering both direction and area, or selecting a viewpoint that is projected near the center of the angle of view. Other methods may also be used, such as

In S301, the grouping unit 102 analyzes the topology of the polygon mesh, and for each polygon, determines adjacent polygons that have the same viewpoint and share a vertex.

Thereafter, the processes of S302 to S304, which will be described later, are repeated until all polygons belong to one of the groups according to the determination process of S305, and the grouping process ends.

In S302, the grouping unit 102 uses an arbitrary polygon that does not belong to any group as a starting point to detect all polygons that do not belong to any group and that are directly or indirectly adjacent to each other and that have the same viewpoint selected. and combine them into one group. The starting polygon and the detected polygons constitute one polygon group. Note that an indirectly adjacent polygon refers to a polygon that is further adjacent to a directly adjacent polygon. That is, a plurality of polygons near the target polygon are grouped into one group.

In S303, the grouping unit 102 projects the polygon group to the selected viewpoint and creates a depth map.

In S304, the grouping unit 102 excludes polygons hidden behind other polygons from the group by occlusion determination using the depth map, and completes the creation of the group. By performing the occlusion determination process, it is possible to prevent multiple polygons from overlapping and corresponding to texels of the texture map.

In this manner, in this embodiment, polygons forming a polygon mesh are grouped to form polygon islands.

Next, the detailed process flow of the polygon island fragment placement process (S204) will be described with reference to the flowchart of FIG.

First, in S400, the placement unit 104 sorts the polygon island fragments in descending order of the area of the circumscribed rectangle, and sets the initial size of the texture map. The size of the texture map is determined by determining the maximum number of pixels in the circumscribed rectangle of the fragment in horizontal and vertical directions, and each is set to a power of 2 that is greater than or equal to the maximum value.

In S401, the placement unit 104 secures a data area for a placed map that manages the placement of fragments on the texture map according to the set size of the texture map, and initializes it to 0 (representing unplaced). The size of the placed map is the same as the texture map.

After that, the placement unit 104 repeats the processes of S402 to S404, which will be described later, until all the fragments are placed on the texture map by the determination process of S405, and ends the fragment placement process.

In S402, the placement unit 104 searches for a location on the texture map where the fragment can be placed. Specifically, the placement unit 104 scans the placed map in raster order from the top left, compares the fragment map with the placed map, and determines whether or not the placed pixels in the placed map overlap with the fragment map. do. If there is no overlap between the placed pixel and the fragment map (if all the values of the placed map corresponding to the fragment map are 0), the placement is determined to be successful, and the scanning ends.

In S403, the placement unit 104 determines the success or failure of fragment placement, and if successful, the process proceeds to S404; if unsuccessful, the process proceeds to S407.

In S407, the placement unit 104 doubles the size of the texture map in the vertical and horizontal directions, returns to S401, and redoes the placement process.

In S404, the placement unit 104 registers the raster position in the fragment, and sets the value of the area of the placed map that overlaps with the fragment map to 1 (representing placement).

In S405, the arrangement unit 104 determines whether all the fragments have been arranged, and if there are any unprocessed fragments, the process returns to S402 and processes the next fragment. On the other hand, if processing of all fragments has been completed, the process advances to S408.

In S408, the arrangement unit 104 assigns texture coordinates on the texture map to each polygon according to the arrangement of the fragments. Specifically, it is calculated by adding the difference between the upper left pixel position of the circumscribed rectangle of the fragment in the selected viewpoint and the texture map to the texture coordinates in the selected viewpoint.

FIG. 10 is a diagram schematically explaining fragment placement processing in this embodiment. Here, a process of arranging each polygon fragment 1001, 1002, 1003, and 1004 shown in FIG. 10(a) on a texture map 1100 as shown in FIG. 10(b) will be described.

First, fragment 1001 is placed on texture map 1100. Next, the fragment 1002 is placed on the texture map 1100 so as not to overlap the placed fragment 1001. Next, fragment 1003 is arranged so as not to overlap the already arranged fragments 1001 and 1002. Finally, fragment 1004 is arranged so as not to overlap the already arranged fragments 1001 to 1003. The fragments are sorted and processed in descending order of area, so the fragments are arranged in descending order of area. Moreover, each fragment is arranged while maintaining its shape and size.

FIG. 10C shows the process of determining overlap using the placed map 1200 when placing the fragment 1004 on the texture map 1100. As illustrated, the fragment 1004 scans the placed map 1200 in raster order from the top left, searching for an area that does not overlap with the area where other fragments have been placed (areas shaded in black).

In this way, in the fragment placement process in this embodiment, each polygon fragment is placed on the texture map, and texture coordinates on the texture map are given to each polygon.

As described above, textured polygon mesh data is generated from a polygon mesh and captured images from multiple viewpoints. In this embodiment, the same viewpoint is selected and adjacent polygons are grouped together to form fragments, and the fragments are placed on the texture map. Furthermore, the formed fragments are placed on the texture map while maintaining their shape and size. According to this embodiment, it is possible to prevent the image quality of a drawn object from deteriorating. Furthermore, the number of pixels of the texture image (ie, texture map) generated at this time can be reduced.

(Hardware configuration)
Referring to FIG. 8, the hardware configuration of the image processing apparatus in this embodiment will be described. The image processing device 800 includes a CPU 801 , a ROM 802 , a RAM 803 , an auxiliary storage device 804 , a display section 805 , an operation section 806 , a communication I/F 807 , and a bus 808 .

The CPU 801 implements each function shown in FIG. 1 by controlling the entire image processing apparatus 800 using computer programs and data stored in the ROM 802 and RAM 803. Note that the image processing apparatus 800 may include one or more dedicated hardware or GPUs (Graphics Processing Units) different from the CPU 801, and at least part of the processing by the CPU 801 may be executed by the dedicated hardware or GPU. Examples of specialized hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 802 stores programs that do not require modification. The RAM 803 temporarily stores programs and data supplied from the auxiliary storage device 804, data supplied from the outside via the communication I/F 807, and the like. The auxiliary storage device 804 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.

The display unit 805 is configured with, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for a user to operate the image processing apparatus 800. The operation unit 806 includes, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and inputs various instructions to the CPU 801 in response to user operations.

A communication I/F 807 is used for communication with an external device of the image processing apparatus 800. For example, when the image processing apparatus 800 is connected to an external device by wire, a communication cable is connected to the communication I/F 807. When the image processing device 800 has a function of wirelessly communicating with an external device, the communication I/F 807 includes an antenna. A bus 808 connects each part of the image processing apparatus 800 and transmits information.

In FIG. 8, it is assumed that the display section 805 and the operation section 806 exist inside the image processing apparatus 800, but at least one of the display section 805 and the operation section 806 exists outside the image processing apparatus 800 as a separate device. You may do so. In this case, the CPU 801 may operate as a display control unit that controls the display unit 805 and an operation control unit that controls the operation unit 806.

<Second embodiment>
In the first embodiment, a method has been described in which all polygon fragments of a polygon mesh are arranged so that they do not overlap on the texture map. However, polygon mesh textures may have repeated patterns or multiple areas with little change. Therefore, in this embodiment, a method will be described in which the number of pixels of a texture image is further reduced by allowing polygon fragments to overlap on a texture map and using the same texture area for a plurality of fragments.

This embodiment differs from the first embodiment in the processing of the placement unit 104 and texture value determination unit 105. In this embodiment, the arrangement unit 104 arranges the fragments and determines the pixel values of the texture map.

FIG. 9 shows a detailed processing flow of fragment placement processing in this embodiment. Here, parts that are different from the processing flow shown in FIG. 4 will be mainly explained. The processes in S900, S903, S905, S907, and S908 in FIG. 9 are the same as the processes in S400, S403, S405, S407, and S408 in FIG. 4, so the description thereof will be omitted.

In this embodiment, in S901, the placement unit 104 secures data areas for both the texture map and the placed map, and initializes them.

In S902, the placement unit 104 searches for a place where a fragment can be placed, similar to S402 in FIG. 4, but in this embodiment, the method for determining whether a fragment can be placed is different. Here, the pixel values of the viewpoint image overlapping the fragment map and the pixel values on the texture map are compared. If the difference in pixel values is smaller than a predetermined threshold, it is determined that the fragment can be placed in that area even if there is an area where the value of the placed map is 1 (indicating placed). As the difference in pixel values, the average of squared differences in pixel values in the fragment map is used. Pixels whose value in the placed map is 0 (indicating that they are not placed) are excluded from the calculation of the pixel value difference. Note that the difference in pixel values is not limited to this, and for example, the maximum value of the absolute values of the differences may be used, or the sum of the absolute values of the differences may be used.

In S904, the arrangement unit 104 registers the raster position on the texture map in the fragment. Next, the placement unit 104 determines pixel values for pixels whose value in the placed map is 0 (indicating that the pixel is not placed). Finally, the placement unit 104 sets the value of the area of the placed map that overlaps the fragment map to 1 (representing placement). In this way, in this embodiment, if the difference in pixel values is smaller than a predetermined threshold in an area where fragments have already been set on the two-dimensional map, a newly set fragment is set overlapping the area where fragments have already been set. can do.

FIG. 11 is a diagram schematically explaining fragment placement processing in this embodiment. Here, similarly to FIG. 10, a process of arranging each polygon fragment 1001, 1002, 1003, and 1004 shown in FIG. 11(a) on a texture map 1100 as shown in FIG. 11(b) will be explained. do. Note that the fragment 1001 and the fragment 1004 have similar textures.

Similar to the first embodiment, the fragments 1001 to 1003 are sequentially arranged on the texture map 1100 so as not to overlap the already arranged fragments. On the other hand, since fragment 1004 has a similar texture to fragment 1001, it is arranged so as to overlap fragment 1001 on texture map 1100.

FIG. 11C shows the process of determining overlap using the placed map 1200 when placing the fragment 1004 on the texture map 1100. As shown in the figure, the fragment 1004 scans the placed map 1200 in raster order from the upper left, and even if the other fragments are placed in an area (area painted in black), the difference in pixel values is If it is smaller than the threshold, it is determined that placement is possible.

As explained above, according to this embodiment, the same region of the texture map can be shared between polygon islands with high texture similarity, and the number of pixels of the texture image can be further reduced compared to the first embodiment. Can be made smaller. This configuration works effectively for objects with large areas and simple textures, such as when the ground is included in the object forming the texture.

Up to this point, a polygon mesh having polygons as constituent elements has been used as an example of the data representation of the three-dimensional shape of the object, but the data representation of the three-dimensional shape is not limited to this. For example, it is applicable to any data that can express the shape of the surface of an object, such as point cloud data whose constituent elements are vertices or volume data whose constituent elements are voxels.

In addition, in a texture map, normal information can be written to express an anisotropic texture, depth information can be written to give the texture map three-dimensional information, and height information can be written to express an anisotropic texture on the object surface. Other information such as adding unevenness may also be written.

(Other embodiments)
The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 viewpoint setting section 101 model acquisition section 102 grouping section 103 fragment forming section 104 arrangement section 105 texture value determining section

Claims (14)

An image processing device that generates a texture map corresponding to three-dimensional shape data of an object and representing at least a color of the object,
For the component constituting the three-dimensional shape data, one viewpoint is selected from among a plurality of viewpoints for the object, and the same viewpoint is selected and a plurality of directly and indirectly adjacent viewpoints of the component are selected. grouping means for grouping the constituent elements into one group and classifying the plurality of constituent elements into groups;
a setting means for setting a mapping obtained by projecting a plurality of components belonging to the group onto the selected viewpoint on a two-dimensional map as an area corresponding to the group;
determining means for determining a pixel value according to the area from the image of the selected viewpoint;
generation means for generating a texture map based on pixel values corresponding to the determined area;
An image processing device comprising: The image processing apparatus according to claim 1, wherein the grouping means selects the one viewpoint according to resolutions of the constituent elements in images from the plurality of viewpoints. 3. The image processing apparatus according to claim 1, wherein the setting means sets the areas on the two-dimensional map based on the order of area of the areas so that they do not overlap with each other. The setting means sets the regions on the two-dimensional map based on the order of the area of the regions, and calculates the difference between the pixel value of the image of the selected viewpoint corresponding to the region and the pixel value of the texture map. 3. The image processing apparatus according to claim 1, wherein if the area is smaller than a predetermined threshold value, the newly set area is set to overlap the already set area. Any one of claims 1 to 4, wherein the grouping means excludes from the group a component that is occluded by another component among the plurality of components classified into the one group. The image processing device according to item 1. 6. The image processing apparatus according to claim 1, wherein the three-dimensional shape data of the object is expressed as a polygon mesh whose constituent elements are polygons. The image processing apparatus according to any one of claims 1 to 5, wherein the three-dimensional shape data of the object is expressed by a point group having vertices as the constituent elements. 6. The image processing apparatus according to claim 1, wherein the three-dimensional shape data of the object is expressed as volume data having voxels as the constituent elements. The image processing apparatus according to any one of claims 1 to 8, wherein the texture map includes depth information for the selected viewpoint. The image processing apparatus according to any one of claims 1 to 9, wherein the texture map includes height information representing unevenness on the surface of the object. The image processing apparatus according to any one of claims 1 to 10, wherein the texture map includes normal information for expressing an anisotropic texture. 12. The image processing apparatus according to claim 1, wherein the setting means generates a two-dimensional map according to a total area of regions corresponding to the group. A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 12. An image processing method for generating a texture map corresponding to three-dimensional shape data of an object and representing at least a color of the object, the method comprising:
For the component constituting the three-dimensional shape data, one viewpoint is selected from among a plurality of viewpoints for the object, and the same viewpoint is selected and a plurality of directly and indirectly adjacent viewpoints of the component are selected. a grouping step of grouping the components into one group and classifying the plurality of components into groups;
a setting step of setting a mapping obtained by projecting a plurality of components belonging to the group onto the selected viewpoint on a two-dimensional map as an area corresponding to the group;
a determining step of determining a pixel value according to the area from the image of the selected viewpoint;
a generation step of generating a texture map based on pixel values corresponding to the determined area;
An image processing device comprising:

Images