CN110765305A - Medium information pushing system and visual feature-based image-text retrieval method thereof - Google Patents
Medium information pushing system and visual feature-based image-text retrieval method thereof Download PDFInfo
- Publication number
- CN110765305A CN110765305A CN201911009482.6A CN201911009482A CN110765305A CN 110765305 A CN110765305 A CN 110765305A CN 201911009482 A CN201911009482 A CN 201911009482A CN 110765305 A CN110765305 A CN 110765305A
- Authority
- CN
- China
- Prior art keywords
- image
- feature
- information
- visual
- text
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 71
- 230000000007 visual effect Effects 0.000 title claims abstract description 64
- 238000005516 engineering process Methods 0.000 claims abstract description 43
- 239000013598 vector Substances 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 10
- 230000005540 biological transmission Effects 0.000 claims description 6
- 238000005259 measurement Methods 0.000 claims description 4
- 230000009286 beneficial effect Effects 0.000 abstract description 7
- 230000004927 fusion Effects 0.000 abstract description 7
- 238000009877 rendering Methods 0.000 description 14
- 238000000605 extraction Methods 0.000 description 7
- 230000007246 mechanism Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000011524 similarity measure Methods 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000006073 displacement reaction Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5838—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5846—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using extracted text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5854—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using shape and object relationship
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5862—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Library & Information Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to the technical field of image-text recognition, and provides a media information pushing system and an image-text retrieval method based on visual characteristics, wherein the image-text retrieval method based on the visual characteristics comprises the following steps: the method comprises the steps of obtaining an image-text sample input by a user, extracting feature information in the image-text sample by adopting a visual image searching technology, searching matched media information from a preset image feature library according to the feature information, and displaying the media information in a preset mode. Through the mode, the method and the device can adopt information retrieval based on visual image content and keyword semantics, combine a picture identification matching technology based on content, an image and text cross-media cross retrieval and matching technology, and use an intelligent search technology as a core to realize search service based on various intelligent terminals, further realize content-based image-text fusion big data intelligent search, are beneficial to industrial integrated upgrading, and exert the maximum value of industry.
Description
Technical Field
The application relates to the technical field of image processing, in particular to a visual feature-based image-text retrieval method and a media information pushing system applying the visual feature-based image-text retrieval method.
Background
With the rapid development of information technology, newspapers have gradually expanded from paper form to electronic form, which greatly facilitates users, but poses great challenges to the traditional media industry. Meanwhile, in order to grasp the opportunity of major industrial policies for the nation and the local to greatly promote the development of the cultural industry, and seize the scientific and technological system high points of the industry, more and more media industries need to realize industrial application in order to realize industrial upgrading and improve competitiveness, and the transformation upgrading of the media industries, the content aggregation of the cultural and industrial industries and the value of content mining are promoted.
Aiming at various defects in the prior art, the inventor of the application provides a media information pushing system and a visual feature-based image-text retrieval method thereof through intensive research.
Disclosure of Invention
The invention aims to provide a media information pushing system and a visual feature-based image-text retrieval method thereof, which can adopt information retrieval based on visual image content and keyword semantics, combine a content-based image recognition matching technology and an image-text cross-media retrieval and matching technology, and take an intelligent search technology as a core to realize search services based on various intelligent terminals, further realize content-based image-text fusion big data intelligent search, are beneficial to industrial integration and upgrading, and exert the maximum value of the industry.
In order to solve the above technical problem, the present application provides a visual feature-based image-text retrieval method, as one embodiment, the visual feature-based image-text retrieval method includes:
acquiring a picture and text sample input by a user;
extracting characteristic information in the image-text sample by adopting a visual image searching technology;
searching matched media information from a preset image feature library according to the feature information;
and displaying the media information in a preset mode.
As an implementation manner, the step of obtaining the image-text sample input by the user specifically includes:
and acquiring an image-text sample input by a user through network transmission, local uploading, instant shooting, instant tool drawing and/or copy keytone drawing.
As an implementation manner, the step of extracting feature information in the image-text sample by using a visual image search technology specifically includes:
and extracting characteristic information including color attributes of image pixels and/or interrelations among the pixels in the image-text sample by adopting a visual image searching technology, wherein an index relation is established between the color attributes of the image pixels or the interrelations among the pixels and the image-text sample.
As an implementation manner, the step of extracting, by using a visual image search technique, feature information including a color attribute of an image pixel and/or a correlation between pixels in the image-text sample specifically includes:
and extracting the characteristic information of the color, texture and shape of the bottom layer characteristic of the image pixel in the image-text sample by adopting a visual image searching technology.
As an implementation manner, the step of extracting feature information of color, texture, and shape of bottom features of image pixels in the image-text sample by using a visual image search technique further includes:
and performing semantic recognition on the color, texture and shape of the bottom layer features of the image pixels through a preset image semantic model to obtain feature information with abstract visual features.
As an implementation manner, the step of searching for matching media information from a preset image feature library according to the feature information specifically includes:
searching a plurality of media information similar to the characteristic information from a preset image characteristic library according to the characteristic information;
calculating the feature similarity of the feature information and the plurality of pieces of media information;
and prioritizing the plurality of pieces of media information according to the feature similarity.
As an embodiment, the step of searching a plurality of pieces of media information similar to the feature information from a preset image feature library according to the feature information specifically includes:
and extracting a key word set of basic outline semantics of the feature information according to a knowledge graph in the technical field, and using the key word set as a semantic feature vector of the image to perform image semantic retrieval.
As an embodiment, the step of calculating the feature similarity between the feature information and the plurality of pieces of media information specifically includes:
adopting semantic feature vectors to perform matching calculation between the image-text sample of the feature information and the image to be matched of any piece of media information, wherein a distance measurement formula for performing matching calculation between the image-text sample of the feature information and the image to be matched of any piece of media information by adopting the semantic feature vectors comprises the following steps:
where P, Q represents any two images and n is the number of keywords.
As an implementation manner, the image-text sample includes an image, a video, and/or a document, the media information includes an image, a video, and/or a document, and the step of searching for matching media information from a preset image feature library according to the feature information specifically includes:
and searching cross-media medium information from an image feature library of a public network, a local area network, a museum, an exhibition hall, social media and/or a peer network according to the feature information.
In order to solve the above technical problem, the present application further provides a media information pushing system, as one embodiment, configured with a processor, where the processor is configured to execute program data to implement the visual feature-based image-text retrieval method as described above.
The media information pushing system and the visual feature-based image-text retrieval method thereof comprise the following steps: the method comprises the steps of obtaining an image-text sample input by a user, extracting feature information in the image-text sample by adopting a visual image searching technology, searching matched media information from a preset image feature library according to the feature information, and displaying the media information in a preset mode. Through the mode, the method and the device can adopt information retrieval based on visual image content and keyword semantics, combine a picture identification matching technology based on content, an image and text cross-media cross retrieval and matching technology, and use an intelligent search technology as a core to realize search service based on various intelligent terminals, further realize content-based image-text fusion big data intelligent search, are beneficial to industrial integrated upgrading, and exert the maximum value of industry.
The foregoing description is only an overview of the technical solutions of the present application, and in order to make the technical means of the present application more clearly understood, the present application may be implemented in accordance with the content of the description, and in order to make the above and other objects, features, and advantages of the present application more clearly understood, the following preferred embodiments are described in detail with reference to the accompanying drawings.
Drawings
Fig. 1 is a schematic flow chart of an embodiment of the visual feature-based image-text retrieval method according to the present application.
Fig. 2A is a schematic structural diagram of an embodiment of a media information push system for implementing the visual characteristic-based image-text retrieval method shown in fig. 1.
Fig. 2B is a schematic structural diagram of another embodiment of the media information delivery system of the present application.
Detailed Description
To further clarify the technical measures and effects taken by the present application to achieve the intended purpose, the present application will be described in detail below with reference to the accompanying drawings and preferred embodiments.
While the present application has been described in terms of specific embodiments and examples for achieving the desired objects and objectives, it is to be understood that the invention is not limited to the disclosed embodiments, but is to be accorded the widest scope consistent with the principles and novel features as defined by the appended claims.
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating an embodiment of a visual characteristic-based image-text retrieval method according to the present application.
It should be noted that the visual feature-based text retrieval method according to this embodiment may include, but is not limited to, the following steps.
And step S101, obtaining an image-text sample input by a user.
And S102, extracting characteristic information in the image-text sample by adopting a visual image searching technology.
And step S103, searching matched media information from a preset image feature library according to the feature information.
And step S104, displaying the media information in a preset mode.
The method and the device can adopt information retrieval based on visual image content and keyword semantics, combine a content-based picture identification matching technology, an image and text cross-media cross retrieval and matching technology, and use an intelligent search technology as a core to realize search service based on various intelligent terminals, further realize content-based image-text fusion big data intelligent search, are beneficial to industrial integrated upgrading, and exert the maximum value of the industry.
For example, the step of obtaining the image-text sample input by the user in the embodiment specifically includes: and acquiring an image-text sample input by a user through network transmission, local uploading, instant shooting, instant tool drawing and/or copy keytone drawing.
It is easy to understand that, in this embodiment, the image-text sample input through network transmission, local upload, instant shooting, instant tool rendering, and/or copy keytone rendering refers to an image-text sample transmitted through a network mail, chat software, or a news media, where the local upload may refer to upload and input an image-text sample stored locally in a system, the instant shooting may refer to an image input after being directly shot through a mobile phone, an intelligent camera, or the like, the instant tool rendering may refer to upload rendering of an image-text through a professional rendering tool, and the copy keytone rendering may refer to an image-text simply rendered according to a key tone color block by a user.
It should be noted that, in this embodiment, the step of extracting the feature information in the image-text sample by using the visual image search technology may specifically include: and extracting characteristic information including color attributes of image pixels and/or interrelations among the pixels in the image-text sample by adopting a visual image searching technology, wherein an index relation is established between the color attributes of the image pixels or the interrelations among the pixels and the image-text sample.
Specifically, the step of extracting, by using a visual image search technique, feature information including a color attribute of an image pixel and/or a correlation between pixels in the image-text sample includes: and extracting the characteristic information of the color, texture and shape of the bottom layer characteristic of the image pixel in the image-text sample by adopting a visual image searching technology.
Further, in this embodiment, the step of extracting the feature information of the color, texture, and shape of the bottom layer feature of the image pixel in the image-text sample by using the visual image search technology further includes: and performing semantic recognition on the color, texture and shape of the bottom layer features of the image pixels through a preset image semantic model to obtain feature information with abstract visual features.
It should be noted that, in this embodiment, the step of searching for matching media information from a preset image feature library according to the feature information may specifically include: searching a plurality of media information similar to the characteristic information from a preset image characteristic library according to the characteristic information; calculating the feature similarity of the feature information and the plurality of pieces of media information; and prioritizing the plurality of pieces of media information according to the feature similarity.
It should be noted that, in this embodiment, the step of searching for a plurality of pieces of media information similar to the feature information from a preset image feature library according to the feature information specifically includes: and extracting a key word set of basic outline semantics of the feature information according to a knowledge graph in the technical field, and using the key word set as a semantic feature vector of the image to perform image semantic retrieval.
In addition, the step of calculating the feature similarity between the feature information and the plurality of pieces of media information according to the present embodiment specifically includes: and performing matching calculation between the image-text sample of the feature information and the image to be matched of any media information by adopting a semantic feature vector, wherein a distance measurement formula for performing matching calculation between the image-text sample of the feature information and the image to be matched of any media information by adopting the semantic feature vector is a similarity calculation formula of the image-text sample of the feature information and the image to be matched of any media information, and comprises the following steps:
in equation one, P, Q represents any two images, n is the number of keywords; the similarity of any two images is calculated by comparing the number of keywords of any two images.
It should be noted that, in this embodiment, the image-text sample includes an image, a video, and/or a document, the media information includes an image, a video, and/or a document, and the step of searching for matching media information from a preset image feature library according to the feature information may specifically include: and searching cross-media medium information from an image feature library of a public network, a local area network, a museum, an exhibition hall, social media and/or a peer network according to the feature information. In practical operation, the characteristic similarity between the characteristic information and the plurality of pieces of media information is calculated, and generally, euclidean distance or inner product distance is used to identify the characteristic similarity.
Referring to fig. 2A, fig. 2A is a schematic diagram illustrating an implementation of a media information push system for implementing the visual characteristic-based image-text retrieval method illustrated in fig. 1 according to the present application.
1. The present embodiment may employ information retrieval based on visual image content and keyword semantics.
For example, the application can implement a content-based image retrieval prototype system by using VC + + as a development environment, as shown in fig. 2A, and is mainly used for verifying feasibility and effectiveness of various feature extraction algorithms. Firstly, extracting the characteristics of an example image of the image-text information submitted by a user, then matching the image-text information with characteristic values in an image characteristic library, and finally returning the retrieval result of the medium information to the user. The key modules of the system of fig. 2A may include a query interface module, a feature extraction module, and a retrieval matching module, and in practical applications, each module may have many specific technologies that can be adopted, and the following mainly discusses the functions of each module and related implementation technologies.
(1) An inquiry interface module;
the query interface module in this embodiment may be configured to provide a relevant query interface of the front-end interface, and the user accesses the image library through the query interface to find an image that meets the requirement, and the search result is returned to the user through the interface. The method for providing the user query by the embodiment can comprise the following steps:
query method one, with example images: that is, the user gives an image similar to the desired image as the query image.
And a second query method, namely drawing a sketch: namely, the user draws the image to be inquired by means of the drawing tool.
And a third query method, which utilizes the retrieval of the dominant hue: the user can set the image color percentages and color distribution information to find images with similar colors and ratios.
(2) A feature extraction module;
the present embodiment addresses the problem that content-based image retrieval can solve, namely, the analysis and representation of image content. The analysis and representation of the image content in the present embodiment may refer to analyzing the color attributes of the pixels of the image and the interrelations among the pixels, so as to obtain a series of numbers or descriptive features, which may describe the content of the image itself to some extent. Then, the image can be indexed by utilizing the characteristics, so that the aim of image retrieval is fulfilled. Thus, the representation of the image content in the present embodiment enables the extraction of image features.
On the other hand, the characteristics of the image comprise the bottom-layer characteristics and the high-layer semantic characteristics of the image, wherein the bottom-layer characteristics are used for describing the characteristics common to the image and mainly comprise color, texture, shape and the like, and the high-layer semantic characteristics are used for describing the content information of the image and are relatively abstract. Therefore, the present application preferably employs feature extraction based on underlying features of the image.
(3) A retrieval matching module;
the query interface module in the embodiment can convert the query request of the user into the query feature vector through the feature extraction module, then call the retrieval matching module to calculate the similarity between each feature in the feature library and the feature of the image to be searched, and return the image required by the user according to the similarity from large to small.
The matching used by the content-based image retrieval system of the present embodiment may be ranked based on similarity retrieval, and the present embodiment may select a plurality of existing suitable similarity metric functions.
For example, in the present embodiment, it is considered that the similarity measure of the image content refers to the similarity between image features, which belongs to an important component of image retrieval research, and the performance of the similarity measure method affects the performance of image retrieval, and meanwhile, the complexity of the similarity measure affects the user response time of image retrieval, so that the image feature vector may be assumed to be an element (an element in the distance space is referred to as a point) in the distance space, and further, the similarity between the image features may be measured by calculating the proximity between the two points.
(4) Extracting image semantics;
in order to solve the problem that the image retrieval based on the content can be based on the matching of the bottom layer features of the image, the similarity matching is not high, the retrieval efficiency is not high, and the subjectivity of the user retrieval cannot be reflected, the semantic identification of the visual features of the image can be realized in the retrieval system, so that the wide user requirements are supported, and the necessary abstraction is carried out on the whole semantic representation and processing process. Specifically, the process of abstraction in the present embodiment can be implemented by establishing an image semantic model and adopting a corresponding technology.
For example, the image meaning model of the present embodiment may include hierarchical architectures such as visual feature semantics, object semantics, spatial relationship semantics, scene semantics, behavior semantics, and emotion semantics.
In this embodiment, the image semantic hierarchy may be further divided into three layers, namely, a bottom feature layer, an object layer, and a concept layer. The meaning of the image semantics depends on the visual features of the bottom layer of the image, the spatial relation from the visual features to the object and the subjective consciousness semantics, and the longitudinal development process is the most important and basic content in the image semantics, so that the semantic model can be established by adopting a hierarchical structure.
In a specific implementation process, pixel detection can be performed by combining the outline semantic information of the image. The embodiment can perform similarity retrieval according to semantic description of the image. In this way, the problem that each user has different descriptions of the images, namely subjectivity, can be solved, and therefore relatively similar theme meanings of the real expression of the users can be obtained.
The semantics of the image in the present embodiment may refer to the interpretation of the meaning of the image. Correspondingly, the simple semantics of the embodiment may be the subject term representation of the image, while the complex semantics are narrative description of the image content, and preferably, the embodiment adopts a marking mode of the simple semantics, namely, the image semantics. According to the embodiment, a key word set representing the basic outline semantics of the image can be extracted according to the knowledge map/professional knowledge in the technical field, and the key word set is used as the semantic feature vector of the image.
As described above, the present embodiment can convert the similarity matching between any two images into the matching between semantic feature vectors, and the similarity matching in practical application can be identified by using euclidean distance, inner product distance, and the like.
2. The embodiment may adopt a content-based picture recognition and matching technology.
The method and the device have the advantages that on the basis of analyzing and researching a line matching method and a ratio matching method in the gray-level related image matching algorithm, matching of two images on horizontal and vertical displacement is achieved respectively by the aid of the two methods, specifically, the gray-level related algorithm can be adopted, matching of the two images on the horizontal and vertical displacement can be achieved, and meanwhile image matching under the condition of rotating around an optical axis can be achieved.
Specifically, the present application may use a block module matching method. In the two matching environments, the implementation method can realize the related algorithm by programming in the Matlab programming environment, and finally obtain accurate matching result medium information by using the conclusions through the matching test of the actual image.
In a specific implementation process, the image matching in the embodiment can be divided into several steps of image input, image preprocessing, matching feature extraction, image matching, result output and the like.
3. The implementation mode of the application can also adopt the cross-media cross-retrieval and matching technology of the images and the texts.
In this way, the method and the device can solve the problem that due to the explosive growth trend of multimedia data, a plurality of heterogeneous multimedia data such as images, videos, documents and the like are greatly emerged in Web, digital libraries and other multimedia applications, and similar semantic expressions exist among the heterogeneous multimedia data.
Simultaneously, this application embodiment can also avoid prior art problem: for example, existing retrieval systems or methods are only directed to retrieving a specific media object, such as an image search tool, and have great limitations in the above applications; on the one hand, they are limited to certain single types of media, such as pure image retrieval methods; on the other hand, they rely only on some specific features of the multimedia data, such as TF × IDF of a keyword or color histogram of an image, wavelet texture features, etc., and it is difficult to provide query results that are semantically related.
In other words, the existing retrieval technology based on a single type of media object cannot meet the new requirements of people for multimedia information query in a large number of applications, and the image and text cross-media cross-retrieval and matching technology can be adopted by the embodiment of the application to effectively solve the problems.
It should be noted that the present embodiment implements "cross-media" search, and is mainly embodied in three aspects:
(1) the retrieval mechanism can be compatible with multimedia data belonging to various different modal types, such as texts, images, videos and the like;
(2) the retrieval mechanism can express and utilize various types of knowledge, including bottom layer characteristics of multimedia data, keywords in a text, hyperlinks between data and the like;
(3) the retrieval mechanism can comprehensively use various retrieval methods, compared with a content-based retrieval method, the retrieval mechanism not only can obtain richer retrieval results, but also can further improve the relevance of the retrieval results by using multi-aspect knowledge as far as possible, and is a very active retrieval mechanism relative to conservation.
The method and the device can adopt information retrieval based on visual image content and keyword semantics, combine a content-based picture identification matching technology, an image and text cross-media cross retrieval and matching technology, and use an intelligent search technology as a core to realize search service based on various intelligent terminals, further realize content-based image-text fusion big data intelligent search, are beneficial to industrial integrated upgrading, and exert the maximum value of the industry.
Referring to fig. 2B, the present application further provides a media information pushing system, which is configured with a processor 21, wherein the processor 21 is configured to execute program data to implement the visual feature-based image retrieval method as described above.
In particular, the processor 21 is configured to obtain a teletext sample of the user input.
The processor 21 is configured to extract feature information in the teletext sample using a visual image search technique.
The processor 21 is configured to search for matched media information from a preset image feature library according to the feature information.
The processor 21 is configured to present the media information in a predetermined manner.
The method and the device can adopt information retrieval based on visual image content and keyword semantics, combine a content-based picture identification matching technology, an image and text cross-media cross retrieval and matching technology, and use an intelligent search technology as a core to realize search service based on various intelligent terminals, further realize content-based image-text fusion big data intelligent search, are beneficial to industrial integrated upgrading, and exert the maximum value of the industry.
For example, the processor 21 in this embodiment is configured to obtain a text sample input by a user, and specifically includes: the processor 21 is configured to obtain an image-text sample input by a user through network transmission, local uploading, instant shooting, instant tool rendering, and/or copy keytone rendering.
It is easy to understand that, in this embodiment, the image-text sample input through network transmission, local upload, instant shooting, instant tool rendering, and/or copy keytone rendering refers to an image-text sample transmitted through a network mail, chat software, or a news media, where the local upload may refer to upload and input an image-text sample stored locally in a system, the instant shooting may refer to an image input after being directly shot through a mobile phone, an intelligent camera, or the like, the instant tool rendering may refer to upload rendering of an image-text through a professional rendering tool, and the copy keytone rendering may refer to an image-text simply rendered according to a key tone color block by a user.
It should be noted that, in this embodiment, the processor 21 is configured to extract the feature information in the image-text sample by using a visual image search technique, and specifically may include: the processor 21 is configured to extract, by using a visual image search technique, feature information including a color attribute of an image pixel and/or an interrelation between pixels in the image-text sample, where an index relationship is established between the image pixel color attribute or the interrelation between pixels and the image-text sample.
Specifically, the processor 21 in this embodiment is configured to extract, by using a visual image search technique, feature information including color attributes of image pixels and/or correlations between pixels in the image-text sample, and specifically includes: the processor 21 is configured to extract feature information of color, texture, and shape of bottom features of image pixels in the image-text sample by using a visual image search technique.
Further, in this embodiment, the processor 21 is configured to extract feature information of a color, a texture, and a shape of a bottom feature of an image pixel in the image-text sample by using a visual image search technique, and further includes: the processor 21 is configured to perform semantic recognition on the color, texture, and shape of the bottom-layer feature of the image pixel through a preset image semantic model, so as to obtain feature information with an abstract visual feature.
It should be noted that, in this embodiment, the processor 21 is configured to search for matching media information from a preset image feature library according to the feature information, and specifically may include: the processor 21 is configured to search a plurality of media information similar to the characteristic information from a preset image characteristic library according to the characteristic information; calculating the feature similarity of the feature information and the plurality of pieces of media information; and prioritizing the plurality of pieces of media information according to the feature similarity.
It should be noted that, in this embodiment, the processor 21 is configured to search a plurality of media information similar to the feature information from a preset image feature library according to the feature information, and specifically includes: the processor 21 is configured to extract a key word set of basic outline semantics of the feature information according to a knowledge graph in the technical field, and use the key word set as a semantic feature vector of the image to perform image semantic retrieval.
In addition, it should be particularly noted that, the processor 21 according to this embodiment is configured to calculate feature similarities between the feature information and the plurality of pieces of media information, and specifically includes: the processor 21 is configured to perform matching calculation between the image-text sample of the feature information and the image to be matched of any piece of media information by using a semantic feature vector, where a distance measurement formula for performing matching calculation between the image-text sample of the feature information and the image to be matched of any piece of media information by using the semantic feature vector is a similarity calculation formula of the image-text sample of the feature information and the image to be matched of any piece of media information, and includes:
in equation two, P, Q represents any two images, and n is the number of keywords; the similarity of any two images is calculated by comparing the number of keywords of any two images.
It should be noted that, in this embodiment, the image-text sample includes an image, a video and/or a document, the media information includes an image, a video and/or a document, and the processor 21 is configured to search for matching media information from a preset image feature library according to the feature information, and specifically may include: the processor 21 is configured to search the cross-media information from the image feature library of the public network, the local area network, the museum, the exhibition hall, the social media and/or the peer network according to the feature information. In practical operation, the characteristic similarity between the characteristic information and the plurality of pieces of media information is calculated, and generally, euclidean distance or inner product distance is used to identify the characteristic similarity.
Furthermore, the present application may also provide a computer-readable storage medium as one embodiment, which is used for storing program data, and when the program data is executed by a processor, the program data implements the visual feature-based image-text retrieval method according to any one of the above embodiments.
Specifically, the visual feature-based image-text retrieval method may be as shown in fig. 1 and fig. 2A, and details are not repeated within a range that a person skilled in the art can comprehensively understand in combination with fig. 1 and fig. 2A and the implementation manner thereof.
The media information pushing system and the image-text retrieval method based on the visual characteristics can adopt information retrieval based on visual image content and keyword semantics, combine a content-based picture identification matching technology and an image-text cross-media cross retrieval and matching technology, and take an intelligent search technology as a core to realize search services based on various intelligent terminals, further realize content-based image-text fusion big data intelligent search, are beneficial to industrial integration and upgrading, and exert the maximum value of the industry.
In the present application, the media information pushing system may be a dedicated server, a media company information system, a network media system, or the like.
Although the present application has been described with reference to a preferred embodiment, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the application, and all changes, substitutions and alterations that fall within the spirit and scope of the application are to be understood as being included within the following description of the preferred embodiment.
Claims (10)
1. A visual feature-based image-text retrieval method is characterized by comprising the following steps:
acquiring a picture and text sample input by a user;
extracting characteristic information in the image-text sample by adopting a visual image searching technology;
searching matched media information from a preset image feature library according to the feature information;
and displaying the media information in a preset mode.
2. The visual-feature-based teletext retrieval method according to claim 1, wherein the step of obtaining a teletext sample input by a user specifically comprises:
and acquiring an image-text sample input by a user through network transmission, local uploading, instant shooting, instant tool drawing and/or copy keytone drawing.
3. The visual-feature-based teletext retrieval method according to claim 1, wherein the step of extracting feature information in the teletext sample using a visual image search technique specifically comprises:
and extracting characteristic information including color attributes of image pixels and/or interrelations among the pixels in the image-text sample by adopting a visual image searching technology, wherein an index relation is established between the color attributes of the image pixels or the interrelations among the pixels and the image-text sample.
4. A visual-feature-based teletext retrieval method according to claim 3, wherein the step of extracting feature information including color attributes of image pixels and/or interrelations between pixels in the teletext sample using a visual image search technique specifically comprises:
and extracting the characteristic information of the color, texture and shape of the bottom layer characteristic of the image pixel in the image-text sample by adopting a visual image searching technology.
5. The visual-feature-based teletext retrieval method according to claim 4, wherein the step of extracting feature information of color, texture and shape of underlying features of image pixels in the teletext sample by using a visual image search technique further comprises:
and performing semantic recognition on the color, texture and shape of the bottom layer features of the image pixels through a preset image semantic model to obtain feature information with abstract visual features.
6. The visual-feature-based teletext retrieval method according to claim 1, wherein the step of searching for matching media information from a preset image feature library according to the feature information specifically comprises:
searching a plurality of media information similar to the characteristic information from a preset image characteristic library according to the characteristic information;
calculating the feature similarity of the feature information and the plurality of pieces of media information;
and prioritizing the plurality of pieces of media information according to the feature similarity.
7. The visual-feature-based teletext retrieval method according to claim 6, wherein the step of searching for a plurality of media information similar to the characteristic information from a preset image feature library according to the characteristic information specifically comprises:
and extracting a key word set of basic outline semantics of the feature information according to a knowledge graph in the technical field, and using the key word set as a semantic feature vector of the image to perform image semantic retrieval.
8. The visual-feature-based teletext retrieval method according to claim 7, wherein the step of calculating the feature similarity between the feature information and the plurality of pieces of media information specifically comprises:
adopting semantic feature vectors to perform matching calculation between the image-text sample of the feature information and the image to be matched of any piece of media information, wherein a distance measurement formula for performing matching calculation between the image-text sample of the feature information and the image to be matched of any piece of media information by adopting the semantic feature vectors comprises the following steps:
where P, Q represents any two images and n is the number of keywords.
9. A visual-feature-based teletext retrieval method according to claim 1, wherein the teletext sample comprises images, videos and/or documents, the media information comprises images, videos and/or documents, and the step of searching for matching media information from a preset image feature library according to the feature information specifically comprises:
and searching cross-media medium information from an image feature library of a public network, a local area network, a museum, an exhibition hall, social media and/or a peer network according to the feature information.
10. A media information push system, characterized in that it is provided with a processor for executing program data to implement a visual-feature-based teletext retrieval method according to any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911009482.6A CN110765305A (en) | 2019-10-23 | 2019-10-23 | Medium information pushing system and visual feature-based image-text retrieval method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911009482.6A CN110765305A (en) | 2019-10-23 | 2019-10-23 | Medium information pushing system and visual feature-based image-text retrieval method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110765305A true CN110765305A (en) | 2020-02-07 |
Family
ID=69333055
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911009482.6A Pending CN110765305A (en) | 2019-10-23 | 2019-10-23 | Medium information pushing system and visual feature-based image-text retrieval method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110765305A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021190115A1 (en) * | 2020-03-25 | 2021-09-30 | 北京沃东天骏信息技术有限公司 | Method and apparatus for searching for target |
| CN114077682A (en) * | 2022-01-19 | 2022-02-22 | 广州拟实网络科技有限公司 | Intelligent recognition matching processing method and system for image retrieval and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101739397A (en) * | 2008-11-17 | 2010-06-16 | 新疆亚奥数码科技有限公司 | MPEG-7-based image retrieving system |
| CN101799806A (en) * | 2009-02-06 | 2010-08-11 | 沈阳工大普日软件技术有限公司 | Chest digital X-ray image retrieval system based on multi-information fusion |
| CN104298749A (en) * | 2014-10-14 | 2015-01-21 | 杭州淘淘搜科技有限公司 | Commodity retrieval method based on image visual and textual semantic integration |
| CN110110116A (en) * | 2019-04-02 | 2019-08-09 | 浙江工业大学 | A kind of trademark image retrieval method for integrating depth convolutional network and semantic analysis |
-
2019
- 2019-10-23 CN CN201911009482.6A patent/CN110765305A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101739397A (en) * | 2008-11-17 | 2010-06-16 | 新疆亚奥数码科技有限公司 | MPEG-7-based image retrieving system |
| CN101799806A (en) * | 2009-02-06 | 2010-08-11 | 沈阳工大普日软件技术有限公司 | Chest digital X-ray image retrieval system based on multi-information fusion |
| CN104298749A (en) * | 2014-10-14 | 2015-01-21 | 杭州淘淘搜科技有限公司 | Commodity retrieval method based on image visual and textual semantic integration |
| CN110110116A (en) * | 2019-04-02 | 2019-08-09 | 浙江工业大学 | A kind of trademark image retrieval method for integrating depth convolutional network and semantic analysis |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021190115A1 (en) * | 2020-03-25 | 2021-09-30 | 北京沃东天骏信息技术有限公司 | Method and apparatus for searching for target |
| CN114077682A (en) * | 2022-01-19 | 2022-02-22 | 广州拟实网络科技有限公司 | Intelligent recognition matching processing method and system for image retrieval and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11461392B2 (en) | Providing relevant cover frame in response to a video search query | |
| Marques et al. | Content-based image and video retrieval | |
| US10831814B2 (en) | System and method for linking multimedia data elements to web pages | |
| US8788529B2 (en) | Information sharing between images | |
| US20170024384A1 (en) | System and method for analyzing and searching imagery | |
| US20100325138A1 (en) | System and method for performing video search on web | |
| US20140207778A1 (en) | System and methods thereof for generation of taxonomies based on an analysis of multimedia content elements | |
| WO2009158135A2 (en) | Statistical approach to large-scale image annotation | |
| Lewis et al. | An integrated content and metadata based retrieval system for art | |
| CN110765305A (en) | Medium information pushing system and visual feature-based image-text retrieval method thereof | |
| CN108304588B (en) | Image retrieval method and system based on k neighbor and fuzzy pattern recognition | |
| Chandrakala et al. | Application of artificial bee colony optimization algorithm for image classification using color and texture feature similarity fusion | |
| WO2010071617A1 (en) | Method and apparatus for performing image processing | |
| Wankhede et al. | Content-based image retrieval from videos using CBIR and ABIR algorithm | |
| CN116756363A (en) | Strong-correlation non-supervision cross-modal retrieval method guided by information quantity | |
| Tang et al. | Tag ranking based on salient region graph propagation | |
| CN111178409B (en) | Image matching and recognition system based on big data matrix stability analysis | |
| Sudha et al. | Reducing semantic gap in video retrieval with fusion: A survey | |
| KR101896177B1 (en) | An Image Searching System Providing Multiplex Result | |
| Xin et al. | Laplacian sift in visual search | |
| Sebastine et al. | Semantic web for content based video retrieval | |
| Mourato et al. | Clip art retrieval using a sketch Tablet application | |
| Yang et al. | Searching for images by video | |
| Vijayarajan et al. | A review on ontology based document and image retrieval methods | |
| Rho | Efficient object-based distributed image search in wireless visual sensor networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200207 |