WO2022257315A1

WO2022257315A1 - Artwork identification method and system based on artificial intelligence, and artwork trading method and system

Info

Publication number: WO2022257315A1
Application number: PCT/CN2021/123968
Authority: WO
Inventors: 李應樵; 马志雄
Original assignee: 万维数码智能有限公司
Priority date: 2021-06-09
Filing date: 2021-10-15
Publication date: 2022-12-15

Abstract

The present invention provides an artwork identification method and system based on artificial intelligence, and an artwork trading method and system using the identification method. The artwork identification method comprises the following steps: a user obtains a plurality of views of an artwork to be identified; a three-dimensional model of said artwork is reconstructed by means of the obtained plurality of views; artwork image analysis is performed on the reconstructed three-dimensional model to give an assessment of the similarity with data of an authentic artwork; the possibility of the authenticity of said artwork is given according to the assessment result, and said artwork is traded online according to the identification result. The identification accuracy is high by using the artwork identification method and system based on artificial intelligence of the present invention; when an artwork of a user completely matches an artwork in a database, it is easy to promote artwork trading, the security of artwork trading is improved, and the time cost is reduced.

Description

Artwork identification method and system based on artificial intelligence and art transaction method and system

technical field

The invention belongs to the field of art identification, in particular to an artificial intelligence-based art identification method and system, and an art transaction method and system.

Background technique

The field of artwork appraisal has always required the accumulation of long-term knowledge and experience. With the development of artificial intelligence technology and the popularization of its application, in order to ensure the accuracy of artwork appraisal, more and more researchers hope to apply artificial intelligence technology to artworks. At the same time, in order to promote transactions and ensure transaction security, more and more research results of artificial intelligence technology are applied to the field of art transactions.

CN110399834A discloses an artificial intelligence-based art feature transfer system and its application. Including, the front-end server is used to upload and obtain the image of the artwork to be authenticated submitted by the user; the auxiliary authentication module is used to perform auxiliary authentication on the image of the artwork to be authenticated uploaded and obtained by the front-end server according to the stored auxiliary authentication algorithm Analyze, obtain the auxiliary identification result corresponding to the artwork image to be identified, and transmit the auxiliary identification result to the background server; the background server is used to upload and obtain the artwork image to be identified by the front-end server and The auxiliary identification results are correspondingly recorded, and stored in the corresponding preset artwork identification database, and at the same time, the auxiliary identification results corresponding to the image of the artwork to be identified are transmitted to the appreciation platform; Displaying the corresponding auxiliary identification results of the image of the artwork to be identified can improve the reliability of identifying the authenticity of the artwork. Wherein, in the auxiliary identification analysis, image preprocessing is performed on the image of the artwork to be identified; for example, the subject in the image of the artwork is identified: model classification is performed on the image of the artwork to be identified after preprocessing; Artwork appraisal database is established, according to the pre-processed artwork image identification to be identified according to the preset identification model therein, the classification of the identification artwork image is identified; the auxiliary identification result is obtained; the auxiliary identification module also needs to control the artwork image Perform auxiliary sensory processing: grayscale the artwork image to obtain a grayscale artwork image; carry out pixel segmentation enhancement on the grayscale artwork image; use edge tracking technology to convert the image to Carry out the background removal of the artwork image after the pixel point of the scene is segmented and enhanced, and assign a value of 0 to the value determined as the background by using the edge tracking; carry out intelligent image position correction on the artwork image after the background removal; The artwork image after the position correction carries out the correction of the pixel point, and first judges whether the position corresponding to the pixel point has pixels in the coordinate position in the upper, lower, left, and right sides in the correction process; Removing useless information, that is, judging whether there is a row or a column whose pixel values are all 0 around the image, and if so, removing the row or column, thereby reducing the size of the image and forming the final image to be identified artwork image. In this method, the image of the artwork to be identified is recorded correspondingly to the auxiliary identification result, and the auxiliary identification result corresponding to the image of the artwork to be identified is transmitted to the appreciation platform.

CN11022689A discloses a Xiyang silverware stamp recognition method based on deep learning. A deep neural network is trained on a large number of manually labeled samples, and the neural network obtains judgments on the place of origin, year and other information by learning stamp features, so that users do not need to spend a lot of energy to identify tiny stamps, and can directly identify the silverware itself. Get a more comprehensive understanding and judgment of your style and appearance.

The development of existing technologies shows that with the development of artificial intelligence technology, artificial intelligence technology is more and more applied to the field of art appraisal. However, the application of AI technology only in the identification field obviously cannot meet the demand for prior art transactions. Therefore, it is necessary to provide an art identification method and system based on artificial intelligence, and use the identification method to carry out art trading methods and systems.

Contents of the invention

The object of the present invention is to provide a method and system for art identification based on artificial intelligence and a method and system for art trading using the art identification system.

An art identification method based on artificial intelligence, comprising the steps of: obtaining multiple views of the artwork to be identified; reconstructing the three-dimensional model of the artwork to be identified through the obtained multiple views; The model performs artwork image analysis to give an assessment of the similarity with the real artwork data; according to the evaluation result, the possibility of authenticity of the artwork to be identified is given; wherein, the reconstructed 3D model is used for artwork image analysis. The step of analyzing to give an assessment of the similarity to the real artwork data further includes: inputting an image containing unique features in a reconstructed 3D model of an image of the artwork to be identified; and processing the image multiple times using a convolutional neural network , and classify the image through a fully connected layer; start the training network at a specific layer: apply a filter to each of the images and output its corresponding feature map, and the output feature map is used as a new image input; The fully connected layer and the ReLU function obtain the possibility that the processed image belongs to the classification category, and obtain an evaluation result.

The art identification method based on artificial intelligence of the present invention also includes other aspects, wherein the step of reconstructing the three-dimensional model of the artwork to be identified through the obtained multiple views includes: the multiple views are obtained from different sides Multiple two-dimensional artwork pictures obtained; Extracting and matching key feature points for each of the multiple two-dimensional artwork pictures; using the epipolar constraint relationship of the matched key feature points to obtain these key points in The three-dimensional coordinates in the coordinate system to establish a sparse point cloud; expand the sparse point cloud to generate a dense point cloud; fill the hollow part of the dense point cloud to perform surface reconstruction, and perform texture mapping so that the two-dimensional space The texture information is mapped to three-dimensional space. Features are extracted through the convolutional layer (CONV); the feature map is obtained through the convolution of the receptive field (filter) and the input image; a deeper feature map is obtained by using multiple convolutional layers. The feature maps obtain low-level features such as simple shapes, mid-level features such as complex features, high-level features such as determining specific pattern shapes, and trainable classifications. Wherein the convolution layer is followed by a ReLU activation function, and the input value output value of the ReLU function is any number between 0 and 1; the ReLU function is followed by a pooling layer, and the feature map of the input is performed by the pooling layer. Compression makes the feature map smaller and extracts the main features. In the step of obtaining the evaluation result, the similarity evaluation result is obtained through a deep neural network model (DNN), wherein according to the position division of different layers of DNN, the internal network layer is divided into three types: input layer, hidden layer and output layer, The first layer is the input layer, the last layer is the output layer, and the middle layer is the hidden layer. It also includes giving a similarity evaluation of 0 to 1 to evaluate the similarity between the artwork to be identified and the authentic one. The closer to 1, the more authentic the artwork to be identified. When the result is 1, the artwork to be identified is more authentic. The artwork is consistent with the authenticity. Wherein the extraction and matching steps of carrying out key feature points each include: extract key feature points based on three conditions: (1) color, (2) shape, (3) pattern; and adopt block-based and SIFT feature-based Matching method, the KD-TREE method is used in the matching process to match the nearest neighbor feature points, and multi-view geometry is used to limit. The step of obtaining the three-dimensional coordinates of these key points in the coordinate system by using the epipolar constraint relationship of the matched key feature points to establish a sparse point cloud includes: a structure from motion (SFM) method, including triangulation, cluster constraints, etc. process, using two scenes or multiple scenes to automatically restore camera motion and scene structure, starting from the acquired series of images of the target object, and finally obtaining a relatively high-precision sparse 3D point cloud of the target object; the SFM method includes incremental, hierarchical or global strategy; and consistency search and incremental reconstruction; where in the consistency search stage, images with overlapping scenes in the data set are found, and images with overlapping scenes are identified where the same target point is on multiple images Projection; in the incremental reconstruction stage, the scene graph obtained earlier is input, and the output is the pose estimation of the image and the three-dimensional coordinate points in space. Correspondingly, the present invention also provides a system for art identification.

The present invention also provides a method for online artwork transaction, including recording the entity certificate of the authenticated authentic artwork to the artwork transaction block chain; users who intend to participate in the artwork transaction enter the artwork transaction website to obtain the target artwork Basic information; the seller obtains the authenticity analysis result of the artwork through the artwork identification method of the present invention; the buyer completes the transaction of the authentic artwork through the smart contract; wherein the entity certificate of the authenticated authentic artwork is recorded in the artwork transaction block The steps of the chain also include: recording the information of the artwork entity certificate and the information of the buyer and the seller into the blockchain through the smart contract; spreading the smart contract through the P2P network in the entire blockchain network; and the buyer completes the transaction through the smart contract. The transaction steps of authentic works of art also include: regularly checking the blockchain, and automatically executing the smart contract to complete the transaction when specific conditions are triggered.

The online artwork trading method of the present invention also includes other aspects, wherein the triggering condition is that the seller's artwork to be traded is determined to be authentic after artwork authenticity analysis, and is suitable for the buyer in matching. The buyer and seller’s information is the identity information of the buyer and the seller, the transaction object, the transaction price and the conditions that need to be triggered for the establishment of the transaction, such as the matching of the transaction location, the deposit payment ratio, the transaction period, the transaction completion standard, etc. The art transaction block chain is a Bitcoin platform, a side chain connected to the Bitcoin main block chain, a public block chain platform NXT including an operating smart contract, and Ethereum. The steps to complete the transaction of authentic works of art through smart contracts include: contract formulation stage; contract programming stage; contract deployment stage; contract trigger stage; blockchain verification stage; and contract execution stage. It also includes: the contract between the buyer and the seller is written into the blockchain in the form of code, and the contract is made public. Correspondingly, the present invention also provides a system for artwork trading.

Using the above-mentioned art identification method and system of artificial intelligence, the identification accuracy is high, and it is easy to facilitate the art transaction under the condition that the user's artwork completely matches the database artwork. Therefore, the art identification method and system based on artificial intelligence of the present invention technically realize the accurate identification of artworks, thereby promoting the security of art transactions and reducing time costs.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following will briefly introduce the drawings that are used in the embodiments. Apparently, the drawings in the following description are only some examples of the present invention, and those skilled in the art can also obtain other drawings according to these drawings without any innovative work.

Fig. 1 is a flow chart of the art identification method of the present invention.

Fig. 2 is a flow chart of utilizing the art identification method of the present invention to realize art transaction.

Fig. 3(a) is a flowchart of the three-dimensional model reconstruction steps in the artwork identification method of the present invention.

Fig. 3 (b) is the schematic diagram that the artwork certificate is recorded in the artwork transaction block chain in the artwork transaction method of the present invention.

Fig. 3(c) is a flow chart of transaction using smart contracts in the art transaction method of the present invention.

Fig. 3(d) is a schematic diagram of the smart contract work in the art transaction method of the present invention.

Fig. 4 is an example diagram of the three-dimensional model reconstruction step in the artwork identification method of the present invention.

Fig. 5 is an example diagram of the motion structure recovery step in the three-dimensional model reconstruction step in the artwork identification method of the present invention.

FIG. 6 is a flow chart of the steps of performing artwork image analysis on the reconstructed 3D model to give similarity evaluation in the artwork identification method of the present invention.

FIG. 7 is an example diagram of model training in the step of performing artwork image analysis on the reconstructed 3D model to give similarity evaluation in the artwork identification method of the present invention.

FIG. 8 is a schematic diagram of model classification in the step of performing artwork image analysis on the reconstructed 3D model in the artwork identification method of the present invention to give a similarity evaluation step.

Fig. 9 is a structural diagram of the art identification system of the present invention.

Fig. 10 is a computer product diagram of the portable or fixed storage unit of the art identification system of the present invention.

Detailed ways

Specific embodiments of the present invention will now be described in conjunction with the corresponding drawings. However, this invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. These embodiments are provided only so that the present invention will be thorough and complete so that those skilled in the art can fully describe the scope of the present invention. Wording used in the detailed description of the embodiments illustrated in the drawings should not limit the invention. The present invention includes all the content related to the art trading method in the invention patent application entitled "Method and System for Art Trading Using Artificial Intelligence" submitted by the same applicant on the same day.

Fig. 1 is a flow chart of the art identification method of the present invention. Before the artwork appraisal, set up artwork database, store each authenticated artwork data in the artwork database, the data described here include the title of artwork, appraisal certificate, appraisal expert, certificate number, identification code , such as the unique information of the artwork such as the two-dimensional code, and also includes the three-dimensional file of the artwork, and the three-dimensional file includes several standard photos including some areas of the artwork. The user provides the artwork that he owns, that is, the information of the artwork to be authenticated to the artwork appraisal system of the present invention, and after the analysis of the system, the artwork to be authenticated and the process stored in the artwork database are obtained. The possibility of authenticity of certified authentic artwork data. In step 101, the user obtains multiple views of the artwork to be identified; in step 102, the three-dimensional model of the artwork to be identified is reconstructed through the obtained multiple views; in step 103, the reconstructed three-dimensional model is reconstructed The artwork image analysis gives an assessment of the similarity with the real artwork data; in step 104, according to the assessment result, the possibility of authenticity of the artwork to be identified is given. The detailed steps of the art identification method are described in detail below.

Fig. 2 is a flow chart of utilizing the art identification method of the present invention to realize art transaction. In step 201, record the entity certificate of the authenticated authentic artwork to the artwork transaction block chain; in step 202, the user who intends to participate in the artwork transaction enters the artwork transaction website to obtain the basic information of the target artwork; in step 203, The seller obtains the authenticity analysis result of the artwork through the artwork identification system; in step 204, the buyer completes the transaction of the authentic artwork through the smart contract.

Fig. 3(b) is a schematic diagram of recording the artwork certificate into the artwork trading block chain in the artwork trading method of the present invention. The main function of blockchain is to store information. Any information that needs to be saved can be written into the blockchain and can also be read from it, so it is a database. Due to the characteristics of the blockchain, anyone can set up a server, join the blockchain network, and become a node. In the blockchain world, there is no central node, each node is equal, and they all save the entire database. You can write/read data to any node, because all nodes will be synchronized at the end to ensure that the blockchain is consistent. In the present invention, the unique confirmation information of authentic works of art, that is, the information 312 of the artwork entity certificate 311, is written into the block chain 316, and the information includes but not limited to identification expert identity information, anti-counterfeiting watermark information, certificate number , certificate issuance date, QR code information, etc. Since each block has a one-to-one correspondence with the hash, the hash of each block is calculated for the "head". That is to say, the characteristic values of the block header are connected together in order to form a very long string, and then the hash is calculated for this string. Because the block header contains a lot of content, including the hash of the current block body and the hash of the previous block. This means that if the content of the current block body changes, or the hash of the previous block changes, it will definitely cause the hash of the current block to change. Each block of the blockchain is connected to the previous block, which ensures its own reliability, because once the data is written, it cannot be tampered with. Therefore, the accuracy and uniqueness of the physical certificate information of authentic works of art are guaranteed.

Fig. 3(c) is a flow chart of transaction using smart contracts in the art transaction method of the present invention. In step 321, the authentic artwork certificate is recorded in the block chain; in step 322, the information of the buyer and the seller is recorded in the block chain, and the artwork to be traded by the seller is determined to be authentic after the analysis of the authenticity of the artwork , and trigger the signing of the smart contract when a suitable buyer is matched. The

information

313 and 314 of the buyer and the seller includes, but is not limited to, the identity information of the buyer and the seller, the transaction target, the transaction price, and the conditions that need to be triggered for the establishment of the transaction, such as the matching of the transaction location, the deposit payment ratio, the transaction period, and the transaction completion standard. The above step of recording the information of the buyer and the seller into the block chain is to record the information 312 of the artwork entity certificate 311 and the

information

313, 314 of the buyer and the seller into the block chain through the smart contract 315 . In step 323, the smart contract 315 is diffused in the entire blockchain network through the P2P network; in step 324, the blockchain is checked regularly, and when a specific condition is triggered, the smart contract is automatically executed to complete the transaction.

Fig. 3(d) is a schematic diagram of the smart contract work in the art transaction method of the present invention. Smart contracts are codes that can automatically execute the "if this happens, do that result" in traditional contracts, thus providing distributed trusted computing. In the blockchain system of the present invention, there is no single source of control. A distributed architecture with a consensus mechanism means that multiple participants are constantly checking and updating the ledger, and any situation that does not meet the pre-agreed rules will be rejected by other participants. After the buyer and seller have agreed on the artwork to be traded, both parties have a copy of the original trade documents (terms and conditions of trade). With smart contracts, there is a set of trade terms agreed upon in advance, written in computer code. Instead, there may be a mismatch or “break” when the parties disagree on the outcome of the transaction due to a variety of factors: mutual misunderstanding of the original trade terms; confusion due to multiple copies of the original trade terms; Or disagree about what actually happens in the external dependencies. In one embodiment, the artwork transaction of the present invention can be realized by utilizing the Bitcoin platform. The condition that Bitcoin can achieve is that multiple signers are required to sign the transaction before payment, for example, two signers are required in a check. In another embodiment, the use of side chains, that is, blockchains connected to the Bitcoin main blockchain, enables smart contract functions: by allowing different blockchains to run in parallel with Bitcoin, and to support Jump between the main chain and the side chain, and the side chain can be used to execute logic. In another implementation, the public blockchain platform NXT is utilized, which includes a series of currently running smart contracts. However it is not Turing complete and must use existing templates. In another embodiment, Ethereum is a public blockchain platform, which is currently the most advanced blockchain supporting smart contracts. Ethereum adopts a "Turing complete" coding system, which allows any logic to be put into Ethereum smart contracts and run by the entire network.

The part 331 of the smart contract utilized in the art transaction method of the present invention includes the following stages: the 3311 stage is the contract formulation stage; the 3312 stage is the contract programming stage; the 3313 stage is the contract deployment stage; the 3314 stage is the contract trigger stage; 3315 Stage 3316 is the contract execution stage. It is also possible to combine the smart contract with the block chain to form the following stages: in the 3321 stage, the contract between the buyer and the seller is written into the block chain in the form of code, and the contract is made public (that is, the above-mentioned step 322); The network spreads across the entire network of the blockchain (that is, the above-mentioned step 323); at the stage 3323, the blockchain is regularly checked, and as long as the specific conditions for the implementation of the art transaction contract are triggered, the contract is automatically executed (that is, the above-mentioned step 324). The contract formulation of the artwork transaction system of the present invention is consistent with the rights and obligations of the general artwork contract transaction contract, the difference lies in the contract trigger stage, as long as the seller’s artwork is evaluated as authentic and the transaction target of the artwork to be traded is agreed In this case, after blockchain verification, the transaction is completed. Instead of repeatedly discussing the terms of the contract. In this way, art transactions can be completed quickly and safely.

Fig. 3 (a) is a flowchart of the three-dimensional model reconstruction steps in the artwork identification method of the present invention. The three-dimensional model reconstruction step in the art identification method is to reconstruct a realistic three-dimensional model of the artwork to be identified through multiple two-dimensional artwork image files. Specifically, it is divided into the following main steps: In step 301, from Obtain a plurality of two-dimensional artwork image files intended to be identified from different sides; in step 302, extract and match key feature points for each of the plurality of two-dimensional artwork image files; in step 303, Using the epipolar constraint relationship of the matched key feature points to obtain the three-dimensional coordinates of these key points in the coordinate system to establish a sparse point cloud; in step 304, in order to further enrich the three-dimensional information, the sparse point cloud is expanded to generate dense points cloud; in step 305, fill the hollow part of the dense point cloud to perform surface reconstruction, and perform texture mapping to map the texture information in the two-dimensional space to the three-dimensional space.

In a specific embodiment of the present invention, the purpose of three-dimensional model reconstruction is to fully extract the characteristics of the artwork to be identified, using the Harris (Harris) corner detection algorithm, which is characterized by calculation and detection, but it is not sensitive to scale changes. More sensitive; or sift operator, that is, to perform scale space extremum detection, the image is convolved with a Gaussian filter at different scales, and then use continuous Gaussian to blur the image difference to find the key point, the algorithm is characterized by Both scale and rotation are invariant, but the calculation is complicated.

Adopt these methods to carry out the extraction and matching of key feature point in each of described multiple two-dimensional artwork picture files; For two-dimensional artwork picture file, extract key feature point based on three conditions: (1) color, ( 2) shape, (3) pattern; used to detect good features and match feature points between every two images, for example, using block-based and SIFT feature-based matching methods, first using KD-TREE in the matching process The method matches the nearest neighbor feature points, and then uses multi-view geometry to limit.

The three-dimensional coordinates of these key points in the world coordinate system are obtained by using the epipolar constraint relationship of matching key points to build a sparse point cloud. The method that can be used is the Structure from Motion (SFM) method, including triangulation, clustering constraints and other processes, using two or more scenes to automatically restore camera motion and scene structure, starting from the acquired series of images of the target object , and finally obtain a high-precision sparse 3D point cloud of the target; the SFM method includes incremental, hierarchical and global methods. Fig. 5(a)-(c) are example diagrams of the motion structure recovery step in the three-dimensional model reconstruction step in the artwork identification method of the present invention. Among them, Fig. 5(a) is an incremental SFM strategy. The incremental method first selects two two-dimensional images for initialization, and then performs registration and triangulation of points one by one. Fig. 5(b) is a hierarchical SFM strategy , the hierarchical type is to group the two-dimensional images first, register each group, and then perform registration and reconstruction on the results of the previous step; Fig. Registration and reconstruction.

The SFM method adopted for 2D artwork photos is divided into two parts: consistency search and incremental reconstruction. Among them, the consistency search is to find images with overlapping scenes in the data set, and identify the projection of the same target point on multiple images in the images with overlapping scenes. The SIFT algorithm is used to extract feature points, and the feature description vector is used to find images with overlapping scenes. The traditional brute force method tests the overlapping scenes of each pair of images. For each feature in one image, find another image according to the feature vector. The time complexity is not allowed in the reconstruction of large scenes; and select the image pairs obtained in the previous step that may have overlapping scenes, and use the Random Sample Consensus (RANSAC) algorithm to estimate the image pairs The homography transformation matrix or the essential matrix or the fundamental matrix between them is used to judge whether there are enough feature matching points to satisfy the mapping relationship, so as to determine whether the image pair is really related, and finally obtain the scene map. Among them, images are graph nodes, and there are edges between related image pairs. In the incremental reconstruction stage, the scene graph obtained earlier is input, and the output is the pose estimation of the image and the three-dimensional coordinate points in space. Initialization, select the position in the scene image where the field of view of multiple cameras overlaps to perform binocular initialization. This kind of initialization has high redundancy, which makes the reconstruction more robust and accurate; image registration, find the solved image in the image that needs to be registered to get The point of the three-dimensional coordinates of the space, using its 2D and 3D information, solves the PNP problem (Perspective-n-Point problem) through the RANSAC algorithm and the minimum pose solver, and obtains the pose of the newly added image; triangulation, that is, forward intersection , the newly registered images can not only observe the existing three-dimensional points in space, but also solve the new three-dimensional points in space; beam adjustment, image registration and triangulation are an interrelated process, and the error of registration will affect the accuracy of triangulation and vice versa. As the registration and triangulation processes are repeated, the cumulative error becomes larger and larger, which affects the final reconstruction result. Bundle adjustment is essentially a nonlinear optimization algorithm, and the back-projection error is selected as the cost function, which involves the camera internal parameters, external parameters and point cloud to be optimized. In the process, in order to increase the robustness of the algorithm and not be affected by outliers, loss functions such as Huber and Tukey are added.

Methods for obtaining sparse point clouds also include the bundler method of using multiple image information to obtain sparse point clouds by Noah Snavely et al. or the Visual SFM method implemented by executable programs with interfaces proposed by Changchang Wu.

In order to enrich the relatively small number of sparse point clouds, enrich the three-dimensional information, and then better describe the three-dimensional scene, the sparse point cloud is expanded to form a dense point cloud. In one embodiment, the method adopted is Yasutaka Furukawa et al. The patch-based 3D reconstruction algorithm (Patch-Based Multi-View Stereo Software, PMVS) method proposed in 2007 (CVPR2007PAMI2010).

Among many existing 3D reconstruction algorithms, the method of sparsely reconstructing the plane can be selected, for example, using the template shape (SFT, Shape from Template) theory proposed by Adrien Bartoli et al. to reconstruct dense and complete objects. From some data (point cloud, picture, 3D contour line, etc.) to reconstruct the 3D realistic 3D model of the object, in the reconstruction process, there will be different processing algorithms for the 3D reconstruction of different data, such as point cloud data There are many reconstruction methods for 3D reconstruction, such as methods based on Delaunay triangulation, Voronoi diagram, implicit surface, etc. In addition, the reconstruction process of 3D models (MarchingCube, RayCast, mesh construction, etc.) and the processing algorithms after 3D model generation include But not limited to 3D mesh simplification, 3D mesh encryption, 3D model surface smoothing, 3D model hole repair, etc., which require a lot of 3D graphics knowledge (from simple algorithm of drawing points and lines to complex volume Rendering algorithms, as well as lighting calculations, material mapping, etc.). In the final implementation, we are committed to improving algorithm efficiency and operating performance, proposing and implementing various three-dimensional data structures (KD tree, octree, etc.) to speed up algorithm realization, and parallel algorithms (OpenMP, MPI, etc.) ), etc., to reduce memory usage and running time in the algorithm implementation process (reduce space complexity and time complexity).

Fig. 4 is an example diagram of the three-dimensional model reconstruction step in the artwork identification method of the present invention. Wherein, input the two-dimensional pictures 401 of multiple angles of the artwork to be tested, and use the above-mentioned three-dimensional model reconstruction method to output the posture estimation of the artwork to be tested and the spatial three-dimensional coordinate map 402, thereby obtaining the color of the artwork to be tested , shape and pattern information.

Fig. 6 is the flow chart of the similarity evaluation step of performing artwork image analysis on the reconstructed three-dimensional model in the artwork identification method of the present invention. In step 601, an image containing unique features in the reconstructed 3D model containing the artwork image is input; in step 602, the image is processed multiple times using a convolutional neural network, and the image is classified through a fully connected layer ; In step 603, start the training network at a specific layer: apply a filter to each of the images and output its corresponding feature map, and the output feature map is a new image input; in step 604, through the fully connected layer and The ReLU function obtains the possibility that the processed image belongs to the classification category; in step 605, multiple images are obtained from the user end, and the trained network obtains the evaluation of the similarity between the artwork to be evaluated and the confirmed authentic work to obtain similarity evaluation results.

FIG. 7 is an example diagram of model training in the step of performing artwork image analysis on the reconstructed 3D model to give similarity evaluation in the artwork identification method of the present invention. The above-mentioned three-dimensional model reconstruction method will be used to output the attitude estimation of the image of the artwork to be tested and a small part of the sample in the space three-dimensional coordinate map 402, and the small picture will be used as the input layer of the above-mentioned step 601, such as in FIG. 7 Input image 701 of flowers.

In step 702, the features are extracted through the convolutional layer (CONV). Taking the input image as 32*323 as an example, 3 is its depth (ie R, G, B), and the convolutional layer is a 5*5 *3 Receptive field 705 (filter), the depth of the receptive field must be the same as the depth of the input image. A 28*28*1 feature map can be obtained by convolving a filter with the input image. For example, two receptive fields (filters) are used to obtain two feature maps; multi-layer convolutional layers are used to obtain deeper feature maps. ; In step 714, start the network at a specific layer, wherein, as described in the above step 603, start the training network at a specific layer: apply a filter to each of the images and output its corresponding feature map, the output feature map Input 715 for a new image; Obtain low-level features 706 (e.g. simple shapes), mid-level features 707 (e.g. complex features) and high-level features (e.g. can be used to determine the shape of flowers) and trainable classification; convolutional layer After that, it is the ReLU activation function. The purpose of using the ReLU activation function is to avoid small changes in the input, resulting in completely different results in the output structure. In order to simulate more subtle changes, the input and output values are not just 0 to 1, but can be 0 Any number between 1 and 1; the ReLU activation function adds nonlinear factors to construct a sparse matrix, that is, sparsity. This feature can remove redundancy in the data and retain the characteristics of the data as much as possible, that is, most of the sparseness is 0. matrix to represent. In fact, this feature is mainly for ReLU, which is max(0,x), because the neural network is constantly calculating repeatedly, and it actually becomes that it is trying to constantly try to express data with a matrix that is mostly 0 feature, the result is better calculation results due to the existence of sparse features; the pooling layer (Pooling) after the ReLU activation function is used, and the pooling layer compresses the input feature map, on the one hand, it makes the feature map smaller and simplifies The computational complexity of the network; on the one hand, feature compression is performed to extract the main features, and the feature vector output by the convolutional layer is reduced by pooling, while improving the result (not easy to overfit).

In step 703, after performing the convolutional layer, ReLU activation function and pooling layer for many times, enter the fully connected layer (FC); in the fully connected layer 710, connect all features 709 (x ₁ , x ₂ , x ₃ .....), through the ReLU activation function 711, enter the fully connected layer 712 again, and send the output value to the classifier (such as the softmax classifier 713). The next most likely is a car, the third least likely to be a tree, and finally the least likely to be a tree. Each node of the fully connected layer is connected to all the nodes of the previous layer, which is used to integrate the features extracted earlier. Due to its fully connected characteristics, the parameters of the general fully connected layer are also the most. In the above embodiment of flowers, the classified results 704 are flowers, cups, cars and trees.

FIG. 8 is a schematic diagram of model classification in the step of performing artwork image analysis on the reconstructed 3D model in the artwork identification method of the present invention to give a similarity evaluation step. According to the photo 801 provided by the user, enter the trained model 802, and obtain the similarity evaluation result 803 of the deep neural network model (DNN). From the position of the different layers of the DNN, the neural network layers inside the DNN can be divided into three categories, Input layer, hidden layer and output layer, generally speaking, the first layer is the input layer, the last layer is the output layer, and the middle layers are all hidden layers. In one embodiment, a similarity evaluation of 0 to 1 is given to evaluate the similarity between the artwork to be identified and the authentic one, the closer to 1, the more authentic, and when the result is 1, the two are consistent. When the identification result is 1, the above-mentioned art transaction method is started.

Fig. 9 is a structural diagram of the art identification system of the present invention. For example, the server 901 for artwork identification. The artwork authentication server includes a processor 910, where the processor can be a general-purpose or special-purpose chip (ASIC/eASIC) or FPGA or NPU, etc., and a computer program product in the form of a memory 920 or a computer-readable medium. Memory 920 may be electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM. The memory 920 has a storage space 930 for program codes for performing any method steps in the methods described above. For example, the storage space 930 for program codes may include respective program codes 931 for respectively implementing various steps in the above methods. These program codes can be read or written into the processor 910 . These computer program products comprise program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. Such a computer program product is typically a portable or fixed storage unit as described with reference to FIG. 10 . Fig. 10 is a computer product diagram of the portable or fixed storage unit of the art identification system of the present invention. The storage unit may have storage segments, storage spaces, etc. arranged similarly to the memory 920 in the server of FIG. 9 . The program code can eg be compressed in a suitable form. Typically, the storage unit includes computer readable code 931', i.e. code readable by, for example, a processor such as 910, which when executed by the server causes the server to perform the steps of the methods described above. These codes, when executed by the server, cause the server to perform the steps of the methods described above.

Reference herein to "one embodiment," "an embodiment," or "one or more embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Additionally, please note that examples of the word "in one embodiment" herein do not necessarily all refer to the same embodiment.

The above description is only used to illustrate the technical solutions of the present invention, and any person skilled in the art can modify and change the above embodiments without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention should be determined by the scope of the claims. The present invention has been described above with reference to examples. However, other embodiments than those described above are equally possible within the disclosed scope of the present invention. The different features and steps of the invention may be combined in other ways than described. The scope of the present invention is limited only by the appended claims. More generally, one of ordinary skill in the art can readily understand that all parameters, dimensions, materials and configurations described herein are for exemplary purposes and actual parameters, dimensions, materials and/or configurations will depend on the particular application or invention Teach the application for which it is used.

Claims

A kind of art identification method based on artificial intelligence, comprises the following steps:

Obtain multiple views of the artwork to be authenticated;

Reconstruct the three-dimensional model of the artwork to be identified through the multiple views obtained;

Perform artwork image analysis on the reconstructed 3D model to give an assessment of the similarity with real artwork data;

According to the evaluation results, the possibility of the authenticity of the artwork to be identified is given;

Wherein, the step of performing artwork image analysis on the reconstructed three-dimensional model to give an assessment of the similarity with real artwork data also includes:

Input images containing unique features in reconstructed 3D models containing images of artworks to be identified;

Using a convolutional neural network to process the image multiple times, and classifying the image through a fully connected layer;

Start the training network at a specific layer: apply a filter to each of said images and output its corresponding feature map, said output feature map is used as a new image input;

The probability that the processed image belongs to the classification category is obtained through the fully connected layer and the ReLU function, and an evaluation result is obtained.
The art identification method according to claim 1, wherein the step of reconstructing the three-dimensional model of the artwork to be identified through the obtained multiple views comprises:

The plurality of views are a plurality of pictures of two-dimensional artwork obtained from different sides;

Carrying out extraction and matching of key feature points for each of the plurality of two-dimensional artwork pictures;

Use the epipolar constraint relationship of the matched key feature points to obtain the three-dimensional coordinates of these key points in the coordinate system to establish a sparse point cloud;

Expanding the sparse point cloud to generate a dense point cloud;

The cavity part of the dense point cloud is filled to perform surface reconstruction, and texture mapping is performed to map the texture information in the two-dimensional space to the three-dimensional space.
The art identification method as claimed in claim 1, further comprising the steps of extracting features by a convolutional layer (CONV);

The feature map is obtained by convolution of the receptive field (filter) and the input image;

Use multiple convolutional layers to get deeper feature maps.
The art identification method as claimed in claim 3, wherein the low-level features such as simple shapes, middle-level features such as complex features and high-level features such as determining specific pattern shapes and trainable features are respectively obtained in the feature map. Classification.
The art identification method as claimed in claim 3, wherein the convolutional layer is followed by a ReLU activation function, and the input value output value of the ReLU function is any number between 0 and 1;

The ReLU function is followed by a pooling layer, which compresses the input feature map to make the feature map smaller and extracts main features.
The art identification method as claimed in claim 1, wherein in the step of obtaining evaluation results, the similarity evaluation results are obtained by a deep neural network model (DNN), wherein according to the position division of different layers of DNN, its internal network layers are divided into There are three types of input layer, hidden layer and output layer, in which the first layer is the input layer, the last layer is the output layer, and the middle layer is the hidden layer.
The art identification method according to claim 6, which also includes giving a similarity evaluation of 0 to 1, evaluating the degree of similarity between the artwork to be identified and the original, the closer to 1, the more authentic the artwork to be identified , when the result is 1, the artwork to be identified is consistent with the authentic one.
The art identification method as claimed in claim 2, wherein each of the extraction and matching steps of carrying out key feature points comprises:

Key feature points are extracted based on three conditions: (1) color, (2) shape, (3) pattern; and

The block-based and SIFT feature-based matching methods are used, and the KD-TREE method is used to match the nearest neighbor feature points in the matching process, and multi-view geometry is used to limit.
The art identification method as claimed in claim 2, wherein the step of obtaining the three-dimensional coordinates of these key points in the coordinate system by using the epipolar constraint relationship of the matched key feature points to set up a sparse point cloud comprises:

Structure from Motion (SFM) method, including triangulation, clustering constraints and other processes, uses two or more scenes to automatically restore camera motion and scene structure, starting from the acquired series of images of the target, and finally obtains a higher-precision target Sparse 3D point cloud;

The SFM method includes incremental, hierarchical, or global strategies;

And consistent search and incremental reconstruction;

Among them, in the consistency search stage, find images with overlapping scenes in the data set, and identify the projection of the same target point on multiple images in images with overlapping scenes;

In the incremental reconstruction stage, the scene graph obtained earlier is input, and the output is the pose estimation of the image and the three-dimensional coordinate points in space.
An art identification system based on artificial intelligence, including:

The view acquisition module obtains multiple views of the artwork to be identified;

The 3D model reconstruction module is used to reconstruct the 3D model of the artwork to be identified through the obtained multiple views;

The similarity assessment module analyzes the reconstructed 3D model for the artwork image to give an assessment of the similarity with the real artwork data;

Authenticity judgment module, which gives the possibility of the authenticity of the artwork to be identified;

Wherein, the similarity evaluation module also includes:

An image input module, which inputs images containing unique features in the reconstructed 3D model containing images of works of art to be identified;

The image classification module utilizes a convolutional neural network to process the image multiple times, and classifies the image through a fully connected layer;

The feature map output module starts the training network at a specific layer: applies a filter to each of the images and outputs its corresponding feature map, and the output feature map is used as a new image input;

The classification possibility determination module obtains the possibility that the processed image belongs to the classification category through the fully connected layer and the ReLU function, and obtains an evaluation result.
The artwork appraisal system according to claim 10, wherein said three-dimensional model reconstruction module, wherein: said multiple views are multiple two-dimensional artwork pictures obtained from different sides; also comprising:

The feature point extraction and matching module is used to extract and match key feature points for each of the multiple two-dimensional artwork pictures;

The sparse point cloud building module uses the epipolar constraint relationship of the matched key feature points to obtain the three-dimensional coordinates of these key points in the coordinate system to build a sparse point cloud;

A dense point cloud expansion module, which expands the sparse point cloud to generate a dense point cloud;

The surface reconstruction and mapping module fills the hollow part of the dense point cloud to perform surface reconstruction, and performs texture mapping to map the texture information in the two-dimensional space to the three-dimensional space.
Artwork identification system as claimed in claim 10, also comprises, feature is extracted by convolutional layer (CONV);

The feature map is obtained by convolution of the receptive field (filter) and the input image;

Use multiple convolutional layers to get deeper feature maps.
The art identification system as claimed in claim 12, wherein said feature map respectively obtains low-level features such as simple shapes, middle-level features such as complex features, and high-level features such as determining specific pattern shapes and trainable Classification.
The artwork appraisal system as claimed in claim 12, wherein the ReLU activation function is behind the convolutional layer, and the input value output value of the ReLU function is any number between 0 and 1;

The ReLU function is followed by a pooling layer, which compresses the input feature map to make the feature map smaller and extracts main features.
The art identification system as claimed in claim 10, wherein in the classification possibility determination module,

The similarity evaluation results are obtained through the deep neural network model (DNN). According to the position division of different layers of DNN, the internal network layers are divided into three types: input layer, hidden layer and output layer. The first layer is the input layer, the hidden layer and the output layer. The last layer is the output layer, and the middle layer is the hidden layer.
The art identification system as claimed in claim 15, which also includes giving a similarity evaluation of 0 to 1, evaluating the degree of similarity between the artwork to be identified and the original, the closer to 1, the more authentic the artwork to be identified , when the result is 1, the artwork to be identified is consistent with the authentic one.
The art identification system according to claim 11, wherein said feature point extraction and matching module comprises:

Key feature points are extracted based on three conditions: (1) color, (2) shape, (3) pattern; and

The block matching module adopts block-based and SIFT feature-based matching. The matching is to use the KD-TREE method to match the nearest neighbor feature points, and use multi-view geometry to limit.
The art identification system according to claim 11, wherein said sparse point cloud building module comprises:

Using the Restoration Structure (SFM) module, including triangulation and clustering constraints, using two or more scenes to automatically restore camera motion and scene structure, starting from the acquired series of images of the target, and finally obtain a high-precision sparse 3D target point cloud;

The SFM module includes incremental, hierarchical, or global strategy modules;

and consistency search and incremental reconstruction modules;

Among them, in the consistency search stage, find images with overlapping scenes in the data set, and identify the projection of the same target point on multiple images in images with overlapping scenes;

In the incremental reconstruction stage, the scene graph obtained earlier is input, and the output is the pose estimation of the image and the three-dimensional coordinate points in space.
A method of online art trading comprising,

Record the physical certificate of the authenticated authentic artwork to the artwork transaction blockchain;

Users who intend to participate in the artwork transaction enter the artwork transaction website to obtain the basic information of the target artwork;

The seller obtains the authenticity analysis result of the artwork through the artwork identification method described in claims 1-9;

Buyers complete the transaction of authentic works of art through smart contracts;

The step of recording the physical certificate of the authenticated authentic artwork to the artwork transaction blockchain also includes:

The information of the artwork entity certificate and the information of buyers and sellers are recorded in the blockchain through smart contracts;

Proliferate smart contracts throughout the blockchain network through the P2P network; and

Among them, buyers complete the transaction steps of authentic works of art through smart contracts, including:

The blockchain is checked regularly, and when certain conditions are triggered, the smart contract is automatically executed to complete the transaction.
The online artwork transaction method according to claim 19, wherein the trigger condition is that the seller's artwork to be traded is determined to be authentic after artwork authenticity analysis, and is suitable for the buyer after matching.
The online art transaction method according to claim 20, wherein the information of the buyer and the seller is the identity information of the buyer and the seller, the transaction target, the transaction price and the conditions that need to be triggered for the establishment of the transaction, such as the matching of the transaction location and the deposit payment ratio , transaction deadline, transaction completion criteria, etc.
The online artwork transaction method according to claim 19, the artwork transaction block chain is a Bitcoin platform, a side chain connected to the Bitcoin main block chain, and a public block comprising an operating smart contract Chain platform NXT, Ethereum.
The online artwork transaction method as claimed in claim 19, wherein the step of completing the transaction of the authentic artwork through the smart contract comprises:

Contract formulation phase; contract programming phase; contract deployment phase; contract triggering phase; blockchain verification phase; and contract execution phase.
The online artwork trading method as claimed in claim 19, further comprising:

The contract between the buyer and the seller is written into the blockchain in the form of code, and the contract is made public.
A system for online art transactions, comprising,

Authentic recording module, which records the physical certificate of the authenticated authentic artwork to the artwork transaction blockchain;

Authentic work information reference module, users who intend to participate in the art transaction enter the art transaction website to obtain the basic information of the target artwork;

Artwork authenticity judging module, the seller obtains the analysis result of artwork authenticity through the artwork appraisal system as described in claims 10-18;

In the transaction module, buyers complete the transaction of authentic works of art through smart contracts; the authentic record module also includes,

The information recording module records the information of the artwork entity certificate and the information of buyers and sellers into the blockchain through smart contracts;

Diffusion module, which diffuses smart contracts in the entire blockchain network through the P2P network; and

Among them, the transaction module also includes:

The contract execution module regularly checks the blockchain, and automatically executes the smart contract to complete the transaction when a specific condition is triggered.
The online art trading system according to claim 25, wherein the triggering condition is that the seller's art to be traded is determined to be authentic after art authenticity analysis, and is suitable for the buyer after matching.
The online art trading system according to claim 26, wherein the information of the buyer and the seller is the identity information of the buyer and the seller, the transaction target, the transaction price and the conditions that need to be triggered for the establishment of the transaction, such as the matching of the transaction location and the deposit payment Ratio, transaction period, transaction completion criteria, etc.
The system of online art transaction as claimed in claim 25, the block chain of described art transaction is a Bitcoin platform, a side chain connected to the main block chain of Bitcoin, and a public area comprising an operating smart contract Block chain platform NXT, Ethereum.
The system for online artwork transaction as claimed in claim 25, wherein the transaction module comprises:

contract formulation module; contract programming module; contract deployment module; contract triggering module; blockchain verification module; and contract execution module.
The system for online artwork transaction as claimed in claim 25, further comprising:

The contract between the buyer and the seller is written into the blockchain in the form of code, and the contract is made public.