WO2024053288A1 - ３次元データリダクション装置、及び非一過性の記録媒体 - Google Patents

３次元データリダクション装置、及び非一過性の記録媒体 Download PDF

Info

Publication number
WO2024053288A1
WO2024053288A1 PCT/JP2023/028115 JP2023028115W WO2024053288A1 WO 2024053288 A1 WO2024053288 A1 WO 2024053288A1 JP 2023028115 W JP2023028115 W JP 2023028115W WO 2024053288 A1 WO2024053288 A1 WO 2024053288A1
Authority
WO
WIPO (PCT)
Prior art keywords
reduction
data
shape data
parts
amount
Prior art date
Application number
PCT/JP2023/028115
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
志織 安江
Original Assignee
株式会社Ｎｔｔドコモ
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社Ｎｔｔドコモ filed Critical 株式会社Ｎｔｔドコモ
Priority to JP2024545492A priority Critical patent/JPWO2024053288A1/ja
Publication of WO2024053288A1 publication Critical patent/WO2024053288A1/ja

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics

Definitions

  • the present disclosure relates to a three-dimensional data reduction device that reduces the amount of three-dimensional shape data representing the shape of a three-dimensional object, and a non-transitory recording medium.
  • shape data representing the shape of a three-dimensional object
  • CG Computer Graphics
  • the amount of shape data is often large, and it may be difficult to handle in glass devices such as AR (Augmented Reality) glasses or computer devices with low processing power such as smartphones.
  • AR Augmented Reality
  • Patent Document 1 discloses a technique for easily creating simplified CG data depending on the application.
  • the reduction rate is the compression rate of shape data, that is, the amount of data reduction obtained by reducing the amount of shape data.
  • the higher the reduction rate the smaller the amount of shape data after reduction, but in the image displayed according to the shape data after reduction, the fine shape of the object surface is lost and the image quality is degraded.
  • the "fine shape of the object surface" will be referred to as object details. Therefore, when reducing shape data, it is necessary to repeat the reduction by changing the reduction rate setting while checking the amount of shape data after reduction and the quality of the image displayed according to the shape data. was sometimes necessary. However, there is a problem in that the larger the amount of shape data, the longer the processing time required for one reduction, and it takes a long time to obtain an optimal reduction result in terms of image quality and data amount.
  • a three-dimensional data reduction device is a reduction device that reduces the amount of three-dimensional shape data representing the shape of a three-dimensional object, and includes the following acquisition unit, recognition unit, and division. and a reduction processing section.
  • the acquisition unit acquires three-dimensional shape data representing the shape of a three-dimensional object.
  • the recognition unit recognizes the type of the object based on the three-dimensional shape data.
  • the decomposition unit divides the three-dimensional shape data into a plurality of subsets each corresponding to a plurality of parts constituting the object, based on the recognized type of the object.
  • the reduction processing unit determines the amount of data reduction of the subset corresponding to each of the plurality of parts based on a table having reduction information representing the amount of data reduction of each of the plurality of subsets corresponding to each of the plurality of reference parts. to reduce the amount of the three-dimensional shape data.
  • a non-transitory recording medium includes the steps of acquiring three-dimensional shape data representing the shape of a three-dimensional object, and determining the type of the object based on the three-dimensional shape data. dividing the three-dimensional shape data into subsets each corresponding to a plurality of parts constituting the object based on the recognized type of the object; reducing the data amount of the three-dimensional shape data by determining the subset data reduction amount corresponding to each of the plurality of parts based on a table having reduction information representing the data reduction amount of each of the plurality of subsets;
  • This is a non-transitory recording medium that records a program that causes a computer to execute a method including:
  • the reduction device and the non-transitory recording medium of the present disclosure it is possible to achieve detailed reduction in the amount of data for each part based on the amount of data reduction set in the table for each part that makes up the object. be able to.
  • the contents of the table according to the purpose of the shape data that is, the application that uses the shape data, trial and error such as repeating the reduction of the shape data while changing the amount of data reduction for each part becomes unnecessary.
  • FIG. 1 is a block diagram illustrating a configuration example of a reduction device 10 according to an embodiment of the present disclosure. It is a figure which shows monster doll OB1 which is an example of a three-dimensional object. It is a figure which shows the example of parts division of monster doll OB1.
  • 3 is a diagram showing an example of a parts table TBL1 stored in a storage device 140 of the reduction device 10.
  • FIG. 2 is a flowchart showing the flow of a reduction method executed by the processing device 150 of the reduction device 10 according to the program PR1. It is a figure which shows the example of a division of the parts of chair OB2 which is an example of a three-dimensional object. It is a figure which shows the example of correction of parts division of monster doll OB1. It is a figure which shows the update example of parts table TBL1. It is a figure showing an example of parts table TBL2.
  • FIG. 1 is a block diagram showing a configuration example of a reduction device 10 according to an embodiment of the present disclosure.
  • the reduction device 10 is a three-dimensional data reduction device that reduces the amount of three-dimensional shape data (hereinafter referred to as shape data) representing the shape of a three-dimensional object.
  • shape data three-dimensional shape data
  • Objects in this embodiment are not limited to real objects in real-world view. Objects in this embodiment may include virtual objects that are displayed superimposed on real space.
  • the shape data in this embodiment is a collection of data representing each primitive when the surface of an object is represented by a collection of primitives such as polygons. Specific examples of data representing a primitive include data representing the coordinates of each vertex of the primitive, and data representing a color to be given to the primitive.
  • reduction of the amount of shape data refers to reducing the number of primitives that make up the surface of an object by replacing multiple adjacent primitives with one larger primitive. Since the number of primitives forming the surface of the object is reduced, the image quality of the image of the object displayed according to the shape data after reduction is lower than that before reduction.
  • shape data to be reduced by the reduction device 10 is shape data of a virtual object displayed superimposed on real space in AR.
  • the shape data reduced by the reduction device 10 is stored in a server device that provides content management services to the AR glasses.
  • the content management service refers to a service that acquires position information indicating the position of the AR glasses in the global coordinate system and distributes virtual object information regarding virtual objects placed around the position indicated by the position information.
  • the virtual object information includes shape data of the virtual object, and information indicating the placement position and orientation of the virtual object.
  • the shape data after reduction by the reduction device 10 is included in the virtual object information.
  • the reduction device 10 divides the shape data to be reduced into a plurality of subsets. Specifically, the three-dimensional object represented by the shape data is composed of a plurality of parts. The reduction device 10 divides the shape data into a plurality of subsets each corresponding to a plurality of parts, and reduces the amount of shape data for each part based on a table having reduction information. The reduction information represents the amount of data reduced by reduction. As shown in FIG. 1, reduction device 10 includes a communication device 110, a display device 120, an input device 130, a storage device 140, a processing device 150, and a bus 160. Each of the communication device 110, display device 120, input device 130, and storage device 140 and the processing device 150 are interconnected by a bus 160 that mediates data exchange. The bus 160 may be configured using a single bus, or may be configured using different buses for each element.
  • the communication device 110 is hardware (transmission/reception device) for communicating with other devices.
  • Other devices that communicate with the reduction device 10 are connected to the communication device 110 by wire or wirelessly.
  • Specific examples of other devices connected to the communication device 110 include a server device that provides a content management service in AR, and a supply device that supplies shape data to be reduced to the reduction device 10.
  • Specific examples of the supply device include a 3D scanner that outputs 3D shape data representing the shape of a 3D object by 3D scanning a 3D object, or a 3D scanner that draws a 3D model imitating the 3D object and then outputs 3D shape data representing the shape of the 3D object.
  • An example is a CAD (Computer-Aided Design) device that outputs shape data based on.
  • the display device 120 is, for example, a liquid crystal display.
  • the display device 120 displays various images under the control of the processing device 150.
  • the input device 130 includes, for example, a keyboard with a plurality of keys (operators) and a pointing device such as a mouse.
  • the input device 130 accepts a user's input operation, such as pressing any of a plurality of keys or dragging and dropping using a pointing device.
  • the input device 130 outputs operation data indicating the content of the received input operation to the processing device 150. Through this output, the contents of the user's input operation are transmitted to the processing device 150.
  • a specific example of an input operation performed on the input device 130 in this embodiment is an instruction to correct the division result when a three-dimensional object represented by shape data to be reduced is divided into a plurality of parts. and the operation of registering reduction information that specifies the reduction of parts.
  • the storage device 140 is a recording medium that can be read by the processing device 150.
  • the storage device 140 may be configured of at least one of ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), and the like.
  • the storage device 140 stores in advance a recognition model MDL, a parts table TBL1, and a program PR1.
  • the recognition model MDL is a discriminator that recognizes the type of three-dimensional object (hereinafter referred to as recognition target object) represented by the shape data input to the reduction device 10.
  • the recognition model MDL may be a classifier that recognizes the type of object by feature matching, or may be a classifier generated by deep learning. Specific examples of deep learning include “YOLO,” “faster RCNN,” and “Mask RCNN.”
  • the classifier that recognizes the type of object by feature value matching will be referred to as a first classifier.
  • the classifier generated by deep learning is called a second classifier.
  • the first discriminator is generated in advance by a device other than the reduction device 10. Specifically, a set of known shape data and a label indicating the type of three-dimensional object represented by the known shape data is set as teacher data. A first classifier is generated by repeating a process in which the feature amount of the three-dimensional object represented by the known shape data is associated with a label for each type of object. An advantage of the first classifier is that the time required for generation is short. Even if there is a large amount of training data, generation can be completed in about a few minutes using the first classifier. A disadvantage of the first classifier is that it may fail to detect an object to be identified.
  • An example of such a case is a case where the shape data of the recognition target object represents the three-dimensional shapes of a plurality of objects.
  • the background area of an image generated by projecting a three-dimensional shape represented by shape data onto an arbitrary plane is relatively large.
  • an image generated by projecting a three-dimensional shape represented by shape data onto an arbitrary plane will be referred to as an "image generated based on shape data.”
  • the second discriminator is generated in advance by a device other than the reduction device 10.
  • training data to which a bounding box and a correct label indicating the name of the recognition target object or part divided by the bounding box are added is used.
  • the bounding box is a rectangular area (box) for dividing the recognition target object or each part that constitutes the recognition target object in an image generated based on the shape data.
  • the second classifier learns the features of the recognition target through multiple learnings using a hierarchical structure.
  • the first advantage of the second classifier is that it can specify the area occupied by the recognition target in an image generated based on the shape data of the recognition target.
  • the second advantage is that it is more resistant to changes due to environmental factors than feature matching.
  • the second discriminator has the following two disadvantages.
  • the first disadvantage is that it takes a certain amount of time to generate a classifier, that is, to complete learning.
  • the second disadvantage is that the probability of misrecognition occurring is higher than that of the first classifier.
  • Erroneous recognition refers to incorrect recognition of the type of recognition target because the recognition target is recognized as a trained object even though the recognition target is not a trained object. Note that although such misrecognition may occur with the first classifier, the probability of occurrence is lower than with the second classifier.
  • the second classifier is employed as the recognition model MDL.
  • the recognition model MDL of this embodiment may be generated by any one of the above-mentioned "YOLO”, “faster RCNN”, and “Mask RCNN” methods.
  • the recognition model MDL may be generated by deep learning, which is different from these three methods.
  • the recognition model MDL in this embodiment it is possible not only to recognize the type of the recognition target object but also to divide the recognition target object into a plurality of parts that constitute the recognition target object.
  • the recognition model MDL divides the recognition target into a plurality of parts by outputting region information and parts information for each part.
  • the object included in the image generated based on the input shape data is composed of a plurality of parts.
  • the region information output for each part indicates the range of the region occupied by the corresponding part among the plurality of parts constituting the object.
  • the parts information indicates the corresponding parts. Specific examples of parts information include a character string representing the name of the part or a serial number assigned to the part.
  • the recognition model MDL in this embodiment is generated by deep learning using training data regarding various dolls such as monsters, robots, and animals.
  • the reduction device 10 can recognize the type of doll based on the shape data representing the shape of the three-dimensional doll, such as a monster, robot, or animal, and can recognize each part of the doll.
  • the shape data can be divided into.
  • each piece of shape data divided for each part will be referred to as a subset.
  • FIG. 2 shows a monster doll OB1.
  • the type of doll OB1 is recognized as a "monster" by using the recognition model MDL.
  • the doll OB1 is divided into a first part P1 and a second part P2.
  • the first part P1 is a part corresponding to the head of the monster
  • the second part P2 is a part corresponding to parts other than the head of the monster.
  • the coordinates of the vertices of each primitive represented by the shape data are converted into coordinates in an image generated based on the shape data.
  • the shape data is divided into two subsets (data representing the shape of the part): a subset corresponding to the first part P1 and a subset corresponding to the second part P2.
  • the recognition model MDL in this embodiment is a classifier that recognizes the type of object belonging to one genre, such as a doll.
  • the recognition model MDL may be a classifier that recognizes the types of objects of multiple genres, such as furniture such as chairs and tables, in addition to dolls.
  • the classifier is generated by machine learning using teacher data regarding furniture in addition to teacher data regarding dolls.
  • FIG. 4 is a diagram showing an example of the parts table TBL1.
  • the parts table TBL1 includes types of objects that can be recognized by the recognition model MDL, a plurality of parts (parts information) that constitute the objects, and reduction information regarding the corresponding parts.
  • Each part stored in the parts table TBL1 is an example of a reference part.
  • a reduction rate is employed as the reduction information.
  • the reduction rate in this embodiment is a value expressed as a percentage of the number of primitives that are reduced, when the total number of primitives that constituted a part before reduction is 100. The larger the reduction rate, the larger the amount of data reduction.
  • the reduction rate of a part is determined based on the importance of the part in the object.
  • the importance of a part is determined according to the ease with which the user's line of sight is focused on the part. In this embodiment, the more easily the user's line of sight is focused on a part, the higher the importance and the lower the reduction rate is set.
  • the degree of importance in this embodiment is set based on the results of an experiment in which the user's gaze time is measured for each part when the user looks at the object.
  • the reduction rate of a part whose gaze time was less than 1 second is set to 80.
  • the reduction rate for parts whose gaze time was 1 second or more and less than 3 seconds is set to 50.
  • the reduction rate of parts for which the gaze time was 3 seconds or more is set to 30.
  • the relationship between the part gaze time and the reduction rate may be set as appropriate depending on the type of application that uses the shape data, such as for AR or home game consoles.
  • the reduction rate may decrease in linear proportion to the length of the gaze time.
  • FIG. 4 shows an example of the parts table TBL1 when the three-dimensional object represented by the shape data is a monster doll.
  • Figure 4 shows that the monster doll is composed of two parts, the "head” and the “parts other than the head,” and that the reduction rate of the "head” is 30. ” is shown to have a reduction rate of 80.
  • the reduction rate for the "head” is set lower than the reduction rate for "portions other than the head.” Therefore, in the image of the monster doll OB1 displayed according to the shape data after reduction, the deterioration of the image quality of the head is smaller than that of the parts other than the head. Since the deterioration in image quality of the head, where the line of sight is likely to be focused, is small, the user is less likely to notice the deterioration in image quality due to reduction. This is the reason why the reduction rate is set lower for parts with higher importance, that is, parts that tend to attract the user's gaze.
  • the reduction information in this embodiment is a reduction rate.
  • the reduction information may be (i) a threshold regarding the number of vertices included in a part, (ii) a threshold regarding the number of primitives constituting the part, or (iii) a threshold regarding the data size of a subset corresponding to the part. good.
  • Specific examples of the threshold regarding the number of vertices included in a part include a recommended value or an upper limit that is predetermined in a service provided using shape data.
  • the difference between the number of vertices represented by the subset before reduction and the threshold representing the reduction information is the difference between the data about the part corresponding to the subset. This is the amount of reduction.
  • the reduction information is set so that the more important the part is, the smaller the amount of reduction of vertices will be.
  • threshold values are often defined as requirements in services provided using shape data. The threshold value may be determined for each part based on a combination of the ease with which the part attracts line of sight and the requirements defined in the service.
  • the processing device 150 includes one or more CPUs (Central Processing Units).
  • the processing device 150 reads the program PR1 from the storage device 140 when the reduction device 10 is powered on.
  • the processing device 150 executes the read program PR1.
  • the processing device 150 functions as the acquisition section 150a, the recognition section 150b, the division section 150c, and the reduction processing section 150d shown in FIG. 1 by executing the program PR1. That is, the acquisition unit 150a, recognition unit 150b, division unit 150c, and reduction processing unit 150d shown in FIG. 1 are software modules realized by operating a computer such as a CPU according to software such as a program. The functions carried out by each of these elements are as follows.
  • the acquisition unit 150a acquires shape data output from the supply device by communicating with the supply device using the communication device 110.
  • the recognition unit 150b Based on the shape data acquired by the acquisition unit 150a, the recognition unit 150b recognizes the type of three-dimensional object represented by the shape data using a recognition model MDL. Furthermore, the recognition unit 150b has a function of managing the recognition model MDL.
  • the management functions include editing teacher data for generating the recognition model MDL and executing machine learning using the teacher data.
  • the dividing unit 150c performs a first dividing process or a second dividing process on one piece of shape data.
  • the first division process is a process of dividing the shape data into a plurality of subsets based on the type of object recognized by the recognition unit 150b.
  • the division unit 150c classifies each primitive into parts by specifying to which part each primitive represented by the shape data belongs based on the region information output from the recognition model MDL. do. Based on this classification, the dividing unit 150c divides the shape data into a plurality of subsets for each part.
  • the second division process is a process that is executed when the recognition unit 150b fails to recognize the type of object.
  • the division unit 150c classifies the plurality of primitives represented by the shape data into corresponding groups, assuming that the plurality of primitives that are adjacent to each other and have the same or similar colors belong to the same part. Based on this classification, the recognition unit 150b divides the shape data into a plurality of subsets each corresponding to a plurality of groups. Note that when a plurality of groups are formed by a plurality of primitives that are adjacent to each other and have similar colors, one group is formed in the following manner. For example, the color of a primitive may be represented by the intensity of each color of R (Red), G (Green), and B (Blue).
  • the difference between the value representing the color intensity for each primitive belonging to the same group and the average value of the value for all primitives belonging to the group is a predetermined difference.
  • Groups are formed such that the threshold value is less than the threshold value.
  • the reduction processing unit 150d generates shape data in which the data amount of each of the plurality of subsets is reduced based on the division result by the division unit 150c and the parts table TBL1.
  • reduction in this embodiment refers to reducing the number of primitives forming the surface of an object by replacing a plurality of adjacent primitives with one larger primitive.
  • the reduction processing unit 150d adds a value of 100 percent (for example, if the reduction rate is 80, 80/100) to the total number of primitives constituting the surface of the part as indicated in the reduction information (reduction rate) of the part. Multiply. Then, the reduction processing unit 150d replaces the number of primitives obtained by multiplication in the manner described above. This replacement reduces the amount of shape data.
  • the reduction processing unit 150d when the reduction processing unit 150d receives an instruction to correct the division results through an input operation on the input device 130, the reduction processing unit 150d corrects the plurality of subsets divided by the division unit 150c (correction processing), and then performs shape correction. Reduce the amount of data.
  • the user corrects the parts division result by moving the Bounding Box like a gizmo based on the GUI, or by adding or deleting the Bounding Box.
  • the reduction processing unit 150d performs a table update process to update the parts table TBL1. Execute and then reduce the amount of shape data.
  • the update contents of parts table TBL1 are determined by the user.
  • the processing device 150 operating according to the program PR1 executes the reduction method shown in FIG. 5.
  • the processing device 150 functions as an acquisition unit 150a.
  • the processing device 150 acquires shape data from the supply device by communicating with the supply device using the communication device 110.
  • the processing device 150 functions as a recognition unit 150b.
  • the processing device 150 can recognize the type of object whose three-dimensional shape is represented by the shape data acquired in the acquisition process SA110, that is, the type of recognition target object using the recognition model MDL. Determine whether or not. If the type of recognition target object can be recognized using the recognition model MDL, the determination result of the first determination process SA120 is "Yes”. Conversely, if the type of recognition target object cannot be recognized using the recognition model MDL, the determination result of the first determination process SA120 is "No (negative)".
  • the processing device 150 functions as the division unit 150c and executes the first division process SA130. If the determination result of the first determination process SA120 is "No", the processing device 150 functions as the division unit 150c and executes the second division process SA140.
  • the monster doll OB1 shown in FIG. 2 is the recognition target object.
  • the recognition model MDL in this embodiment is generated by deep learning using training data regarding various dolls such as monsters, robots, and animals. Therefore, if the monster doll OB1 shown in FIG. 2 is a recognition target, the processing device 150 can recognize that the type of the recognition target is a "monster" using the recognition model MDL. Therefore, the determination result of the first determination process SA120 is "Yes", and the first division process SA130 is executed.
  • the shape data representing the three-dimensional shape of the recognition target corresponds to each of the first part P1 and the second part P2 shown in FIG. Divided into subsets.
  • the determination result of the first determination process SA120 is "No"
  • the second division process SA140 is executed.
  • the shape data is divided into a plurality of subsets based on the color of the parts. For example, as shown in FIG. 6, the chair OB2 is divided into a first part P1 corresponding to the backrest, a second part P2 corresponding to the seat plate, and a third part P3 corresponding to the legs. As a result, the shape data of chair OB2 is divided into three subsets corresponding to these three parts, respectively.
  • the processing device 150 functions as a reduction processing unit 150d.
  • the processing device 150 causes the display device 120 to display an image representing the division result of the recognition target, and inquires of the user whether correction of the division result is necessary. Upon receiving the user's response, the processing device 150 determines whether or not the division result needs to be corrected.
  • the determination result of the second determination process SA150 is "Yes”.
  • the processing device 150 executes the third determination process SA170 after executing the correction process SA160.
  • the determination result of the second determination process SA150 is "No". In this case, processing device 150 executes third determination process SA170 without executing correction process SA160.
  • the monster doll OB1 shown in FIG. 2 is the recognition target object, and the shape data of the recognition target is divided into a subset corresponding to the head and a subset corresponding to parts other than the head, as shown in FIG.
  • the processing device 150 causes the display device 120 to display, for example, the image shown in FIG. 3 as an image indicating the division result.
  • FIG. 3 two types of rectangles indicate that the monster doll OB1 has been divided into two parts. One of the two parts is the first part P1 corresponding to the head, and is indicated by a rectangle with a dashed dotted line.
  • the other part is a second part P2 corresponding to a part other than the head, and is indicated by a rectangle with a chain double-dashed line.
  • correction processing SA160 As shown in FIG. 7, correction is performed to divide the parts into a first part P1 corresponding to the head, a second part P2 corresponding to the body, and a third part P3 corresponding to the tail. Assume that the instruction is given by the user.
  • the processing device 150 functions as a reduction processing unit 150d.
  • the processing device 150 determines whether all the reduction information of the parts corresponding to each subset is registered in the parts table TBL1. If the reduction information of all parts is registered in the parts table TBL1, the determination result of the third determination process SA170 is "Yes”. On the other hand, if there is at least one part whose reduction information is not registered in the parts table TBL1, the determination result of the third determination process SA170 is "No".
  • the processing device 150 executes the reduction process SA190 after executing the table update process SA180. On the other hand, if the determination result of the third determination process SA170 is "Yes", the processing device 150 executes the reduction process SA190 without executing the table update process SA180.
  • the processing device 150 functions as the reduction processing unit 150d, displays on the display device 120 a screen that prompts the user to register reduction information regarding parts for which reduction information is not registered, and performs input operations on the input device 130. Accordingly, reduction information regarding the corresponding part is registered in the parts table TBL1.
  • the parts table TBL1 shown in FIG. 7 Since the reduction information (that is, the reduction information corresponding to the tail of the monster doll) is not registered, the determination result of the third determination process SA170 is "No". In this case, table update processing SA180 is executed. In the table update process SA180, reduction information for the tail of the monster doll is additionally registered in the parts table TBL1 in response to an input operation on the input device 130. As a result, parts table TBL1 is updated as shown in FIG.
  • processing device 150 functions as reduction processing section 150d.
  • processing device 150 determines a reduction rate for each part based on parts table TBL1.
  • the processing device 150 reduces the data amount of each subset based on the reduction rate of each part, and generates reduced shape data.
  • This reduced shape data is stored in a server device that provides a content distribution service to the AR glasses, and is transmitted from the server device to the AR glasses.
  • the AR glasses display the image of the monster doll OB1 superimposed on the real space according to the reduced shape data.
  • the overall amount of shape data transmitted from the server device to the AR glasses is reduced, and no particular problem occurs in data processing performed by the AR glasses.
  • the data amount of the subset corresponding to each part is reduced at a different reduction rate for each part of the monster doll OB1.
  • the reduction rate of the head is set lower than the reduction rate of the body and the tail. Therefore, when the image of the monster doll OB1 is displayed on the AR glasses, the details of the head, which tends to attract the user's gaze, are not significantly impaired compared to the details of the body and the tail. Further, details of each of the body and tail are not completely omitted.
  • the present embodiment it is possible to finely reduce the amount of data for each part constituting an object according to the importance of the part. As a result, it is possible to avoid major loss of detail in parts with high importance. It is also possible to avoid completely omitting parts of low importance.
  • the contents of the parts table TBL1 are set appropriately according to the type of application that uses shape data, such as for AR or home game consoles, the reduction of the shape data is repeated while changing the reduction rate of each part. There is no need to go through trial and error.
  • the best method for the application in terms of image quality and data amount depending on the application that uses the shape data. Reduction results can be obtained without trial and error.
  • the recognition model MDL in the above embodiment is a discriminator that recognizes whether a three-dimensional object represented by shape data is a monster, a robot, or an animal doll. For example, when shape data representing the shape of a three-dimensional object belonging to a different genre than dolls, such as a chair or a table, is input, the above-mentioned erroneous recognition may occur. In order to reduce the occurrence of such misrecognition, the first determination process SA120 may be modified as follows.
  • the processing device 150 determines whether the genre of the object whose type can be recognized by the recognition model MDL matches the genre to which the three-dimensional object represented by the input shape data belongs. The user is asked whether or not to do so. In response to this inquiry, if an input operation indicating a match is performed on the input device 130, the processing device 150 recognizes the type of object using the recognition model MDL. On the other hand, if an input operation indicating that they do not match is performed on the input device 130, the processing device 150 sets the determination result as "No" and ends the first determination process SA120. According to this aspect, when the three-dimensional object represented by the shape data acquired in the acquisition process SA110 is not an object of a genre whose object type can be recognized by the recognition model MDL, the object type is The occurrence of misrecognition can be reduced.
  • the first division process SA130 or the second division process SA140 is executed depending on the recognition result of the object type using the recognition model MDL.
  • the recognition model MDL if the type of object cannot be recognized by the recognition model MDL, manual division processing by the user may be performed in place of the second division processing SA140, similar to the correction processing SA160.
  • the second division process SA140 may be always executed without recognizing the type of object using the recognition model MDL, and in this case, the first determination process SA120 and the first division process SA130 are unnecessary.
  • the server device that provides a content distribution service to AR glasses and the reduction device 10 are separate devices, and the shape data that has been reduced by the reduction device 10 is stored in the server device.
  • the server device and the reduction device 10 may be an integrated device. In this case, the server device may store the shape data before reduction and reduce the amount of shape data each time it is distributed to the AR glasses.
  • the server device acquires line-of-sight information indicating the position of the user's line of sight from the AR glasses, and displays an image of the object on the AR glasses.
  • the reduction information may be set depending on which part the user's line of sight is located at when the user's line of sight is located.
  • the server device that also has the functions of the reduction device 10 has a parts table TBL2 shown in FIG. 9 instead of the parts table TBL1 shown in FIG. 4.
  • the reduction information corresponding to "line of sight ON" indicates the reduction rate when the user's line of sight is located
  • the reduction information corresponding to "line of sight OFF" indicates the reduction rate when the user's line of sight is not located. .
  • the processing device of the server device sets a reduction rate depending on whether the user's line of sight is located for each part by referring to the line of sight information and parts table TBL2, and calculates the data of the shape data.
  • Reduce quantity For example, when the image of the monster doll OB1 shown in FIG. , and the reduction rate of the subset corresponding to the torso is determined to be 50.
  • the server device then delivers the reduced shape data to the AR glasses.
  • the program PR1 is stored in the storage device 140 of the reduction device 10, but the program PR1 may be manufactured or sold separately.
  • the program PR1 can be provided to the purchaser by writing the program PR1 on a computer-readable recording medium such as a flash ROM and distributing it, or by downloading it via a telecommunications line such as the Internet.
  • a computer-readable recording medium such as a flash ROM
  • the recognition model MDL and the parts table TBL1 are stored in the storage device 140, but either one or both of the recognition model MDL and the parts table TBL1 can be stored via a telecommunication line.
  • the information may be stored in a storage device that can be accessed by the processing device 150.
  • the importance of each part is set according to the ease with which the user's line of sight is focused on the part, but the importance of each part is set regardless of the ease with which the user's line of sight is focused on the part. Good too.
  • the importance of each part may be set based on the degree to which the creator of the shape data considers it important.
  • the acquisition section 150a, the recognition section 150b, the division section 150c, and the reduction processing section 150d in the above embodiment are all software modules.
  • any one, any two, any three, or all of the acquisition unit 150a, the recognition unit 150b, the division unit 150c, and the reduction processing unit 150d may be a hardware module.
  • Specific examples of the hardware module include DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field Programmable Gate Array), and the like. Even if at least one of the acquisition section 150a, the recognition section 150b, the division section 150c, and the reduction processing section 150d is a hardware module, the same effects as in the above embodiment can be achieved.
  • ROM, RAM, etc. are exemplified as the storage device 140, but the storage device 140 can be a flexible disk, a magneto-optical disk (for example, a compact disk, a digital versatile disk, a Blu-ray (registered) trademark), smart cards, flash memory devices (e.g. cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppy disks, magnetic strips, databases. , a server, or other suitable storage medium.
  • the information, signals, etc. described may be represented using any of a variety of different technologies.
  • data, instructions, commands, information, signals, bits, symbols, chips, etc. may refer to voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may also be represented by a combination of
  • the input/output information may be stored in a specific location (for example, memory) or may be managed using a management table. Information etc. to be input/output may be overwritten, updated, or additionally written. The output information etc. may be deleted. The input information etc. may be transmitted to other devices.
  • the determination may be made based on a value represented by 1 bit (0 or 1), or may be made based on a truth value (Boolean: true or false). , may be performed by numerical comparison (for example, comparison with a predetermined value).
  • each function illustrated in FIG. 1 is realized by an arbitrary combination of at least one of hardware and software.
  • the method for realizing each functional block is not particularly limited. That is, each functional block may be realized using one physically or logically coupled device, or may be realized using two or more physically or logically separated devices directly or indirectly (e.g. , wired, wireless, etc.) and may be realized using a plurality of these devices.
  • the functional block may be realized by combining software with the one device or the plurality of devices.
  • the programs exemplified in the embodiments described above are instructions, instruction sets, codes, code segments, software, firmware, middleware, microcode, hardware description language, or other names. Should be broadly construed to mean program code, program, subprogram, software module, application, software application, software package, routine, subroutine, object, executable, thread of execution, procedure, function, etc.
  • software, instructions, information, etc. may be sent and received via a transmission medium.
  • a transmission medium For example, if the software uses wired technology (coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), etc.) and/or wireless technology (infrared, microwave, etc.) to create a website, When transmitted from a server or other remote source, these wired and/or wireless technologies are included within the definition of transmission medium.
  • wired technology coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), etc.
  • wireless technology infrared, microwave, etc.
  • the information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values from a predetermined value, or other corresponding information. It may also be expressed as
  • the mobile device includes a mobile station (MS).
  • MS mobile station
  • a mobile station is defined by a person skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless It may also be referred to as a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology. Further, in the present disclosure, terms such as “mobile station,” “user terminal,” “user equipment (UE),” and “terminal” may be used interchangeably.
  • connection refers to direct or indirect connections between two or more elements. Refers to any connection or combination and may include the presence of one or more intermediate elements between two elements that are “connected” or “coupled” to each other.
  • the bonds or connections between elements may be physical, logical, or a combination thereof.
  • connection may be replaced with "access.”
  • two elements may include one or more electrical wires, cables, and/or printed electrical connections, as well as in the radio frequency domain, as some non-limiting and non-inclusive examples. , electromagnetic energy having wavelengths in the microwave and optical (both visible and non-visible) ranges, and the like.
  • determining and “determining” used in this disclosure may encompass a wide variety of operations.
  • “Judgment” and “decision” include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, and inquiry. (e.g., searching in a table, database, or other data structure), and regarding an ascertaining as a “judgment” or “decision.”
  • judgment and “decision” refer to receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, and access.
  • (accessing) may include considering something as a “judgment” or “decision.”
  • judgment and “decision” refer to resolving, selecting, choosing, establishing, comparing, etc. as “judgment” and “decision”. may be included.
  • judgment and “decision” may include regarding some action as having been “judged” or “determined.”
  • judgment (decision) may be read as “assuming", “expecting", “considering”, etc.
  • a three-dimensional data reduction device reduces the amount of three-dimensional shape data representing the shape of a three-dimensional object, and includes the following acquisition section, recognition section, division section, and reduction processing section. and has.
  • the acquisition unit acquires three-dimensional shape data representing the shape of a three-dimensional object.
  • the recognition unit recognizes the type of the object based on the three-dimensional shape data.
  • the dividing unit divides the three-dimensional shape data into a plurality of subsets each corresponding to a plurality of parts constituting the object, based on the recognized type of the object.
  • the reduction processing unit determines the amount of data reduction of the subset corresponding to each of the plurality of parts based on a table having reduction information representing the amount of data reduction of each of the plurality of subsets corresponding to each of the plurality of reference parts. to reduce the amount of the three-dimensional shape data.
  • the amount of data reduction is determined for each part constituting an object, and it is possible to achieve fine-grained reduction of the amount of shape data for each part.
  • the contents of the table according to the purpose of the shape data, that is, the application that uses the shape data, it is possible to eliminate the trial and error process of repeating shape data reduction while changing the amount of data reduction for each part. , it is possible to obtain a reduction result that is optimal for the application in terms of image quality and data amount.
  • the data reduction amount of each of the plurality of subsets indicated by the reduction information is It may be set based on the degree of ease of gathering. According to the 3D data reduction device of the second aspect, it is possible to reduce the amount of data for each part by finely setting the amount of data reduction according to the ease of attracting the line of sight for each part constituting the object. This makes it difficult to notice the deterioration in the image quality of the image displayed according to the shape data after reduction.
  • the reduction processing unit corrects the plurality of divided subsets according to user instructions, and corrects each of the plurality of parts.
  • the amount of data of the three-dimensional shape data may be reduced by determining the data reduction amount of the subset. According to the three-dimensional data reduction device of the third aspect, it is possible to reduce the amount of data for each part by correcting the division results by the division section.
  • a non-transitory recording medium acquires three-dimensional shape data representing the shape of a three-dimensional object, and recognizes the type of the object based on the three-dimensional shape data. and dividing the three-dimensional shape data into subsets each corresponding to a plurality of parts constituting the object based on the recognized type of the object; reducing the data amount of the three-dimensional shape data by determining a subset data reduction amount corresponding to each of the plurality of parts based on a table having reduction information representing the data reduction amount of each of the plurality of subsets;
  • This is a non-transitory recording medium that records a program that causes a computer to execute a method including.
  • the amount of data reduction is determined for each part constituting the object, and a detailed reduction in the amount of shape data is achieved for each part. be able to.
  • the contents of the table according to the purpose of the shape data, that is, the application that uses the shape data, it is possible to eliminate the trial and error process of repeatedly reducing the shape data while changing the amount of data reduction for each part. , it is possible to obtain a reduction result that is optimal for the application in terms of image quality and data amount.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)
PCT/JP2023/028115 2022-09-07 2023-08-01 ３次元データリダクション装置、及び非一過性の記録媒体 WO2024053288A1 (ja)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2024545492A JPWO2024053288A1 (enrdf_load_stackoverflow) 2022-09-07 2023-08-01

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022-142147 2022-09-07
JP2022142147 2022-09-07

Publications (1)

Publication Number Publication Date
WO2024053288A1 true WO2024053288A1 (ja) 2024-03-14

Family

ID=90190907

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/028115 WO2024053288A1 (ja) 2022-09-07 2023-08-01 ３次元データリダクション装置、及び非一過性の記録媒体

Country Status (2)

Country Link
JP (1) JPWO2024053288A1 (enrdf_load_stackoverflow)
WO (1) WO2024053288A1 (enrdf_load_stackoverflow)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11339061A (ja) * 1998-05-22 1999-12-10 Fujitsu Ltd 階層化ポリゴンデータを用いた３次元ポリゴン表示装置
WO2020004013A1 (ja) * 2018-06-25 2020-01-02 ソニー株式会社 画像処理装置および画像処理方法
JP6799883B1 (ja) * 2020-07-27 2020-12-16 株式会社Ｖｒｃ サーバ及び情報処理方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11339061A (ja) * 1998-05-22 1999-12-10 Fujitsu Ltd 階層化ポリゴンデータを用いた３次元ポリゴン表示装置
WO2020004013A1 (ja) * 2018-06-25 2020-01-02 ソニー株式会社 画像処理装置および画像処理方法
JP6799883B1 (ja) * 2020-07-27 2020-12-16 株式会社Ｖｒｃ サーバ及び情報処理方法

Also Published As

Publication number Publication date
JPWO2024053288A1 (enrdf_load_stackoverflow) 2024-03-14

Similar Documents

Publication Publication Date Title
US20240252929A1 (en) Systems and methods for 3d scene augmentation and reconstruction
US10049308B1 (en) Synthesizing training data
CN112492388B (zh) 视频处理方法、装置、设备以及存储介质
CN112164002B (zh) 人脸矫正模型的训练方法、装置、电子设备及存储介质
US12272002B2 (en) Method and apparatus for generating virtual character, electronic device and readable storage medium
CN111598818A (zh) 人脸融合模型训练方法、装置及电子设备
US20230091423A1 (en) Method of displaying image, electronic device and storage medium
US20230115887A1 (en) Digital twin sub-millimeter alignment using multimodal 3d deep learning fusion system and method
CN114092615B (zh) 使用人工智能在3d物体上进行uv映射
US20220084271A1 (en) Image processing method and device
US12136192B2 (en) Image processing apparatus
CN111768468A (zh) 图像填充方法、装置、设备及存储介质
EP4462349A1 (en) Data generation method and apparatus, device, and storage medium
CN110110829A (zh) 一种二维码处理方法及装置
AU2017202711A1 (en) Illumination-guided example-based stylization of 3d scenes
CN111177167A (zh) 增强现实的地图更新方法、装置、系统、存储与设备
CN114792355A (zh) 虚拟形象生成方法、装置、电子设备和存储介质
Wang et al. LOD map-a visual interface for navigating multiresolution volume visualization
CN111932438A (zh) 图像风格迁移方法、设备及存储装置
CN114708374A (zh) 虚拟形象生成方法、装置、电子设备和存储介质
US11443485B2 (en) Rendering device, rendering method, 3D model generating method, neural network model generating method, and non-transitory computer readable medium
CN111914106B (zh) 纹理与法线库构建方法、纹理与法线图生成方法及装置
WO2024053288A1 (ja) ３次元データリダクション装置、及び非一過性の記録媒体
CN117689782B (zh) 一种生成海报图像的方法、装置、设备及存储介质
AU2017202712A1 (en) Illumination-guided example-based stylization of 3d scenes

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23862831

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2024545492

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE