KR20210076110A - Methods for finding image regions, model training methods and related devices - Google Patents

Abstract

The present disclosure provides a method for finding an image region, the method comprising: generating (102) a set of region semantic information according to a set of image candidate regions of an image to be found; obtaining (103) an enhanced semantic information set corresponding to the region semantic information set by using the GCN, wherein the GCN is configured to establish an association relationship between the various region semantic information; obtaining (105) a degree of matching between a set of text features corresponding to the text to be found and each enhanced semantic information using the image region finding network model; and determining (106) a target image candidate region from the set of image candidate regions according to the degree of matching between the text feature set and each enhanced semantic information. The present disclosure further discloses a model training method and related apparatus. In the present disclosure, the semantic representation between image candidate regions is improved by using GCN, which can improve image comprehension ability by improving the accuracy of finding image candidate regions.

Description

Methods for finding image regions, model training methods and related devices

This application claims the priority of Chinese Patent Application No. 201910190207.2 ('Method for Finding Image Area, Model Training Method and Related Apparatus'), filed on March 13, 2019, which is hereby incorporated by reference in its entirety Included.

Embodiments of the present disclosure relate to a method for finding an image region, a model training method, and a related apparatus.

As artificial intelligence advances, finding regions in images that correspond to natural sentences is becoming an important task in machine learning. When there are many images, it is usually time-consuming to artificially extract the regions associated with natural sentences, and there is a high probability of errors. Therefore, it is very necessary to use a machine to find the image area.

At present, in a method for finding an image region, a plurality of candidate regions from an image are first extracted by an object proposal method, and then each object proposal and natural sentence are further selected as a target image region to select a local region that best fits the natural language as a target image region. The matching model is used to determine the matching relationship between the two, thereby completing the task of finding the corresponding natural sentence image.

According to a first aspect of the present disclosure, there is provided a method for finding an image region,

generating a set of region semantic information according to the set of image candidate regions of the to-be located image, wherein each region semantic information of the set of region semantic information is a set of image candidate regions. Corresponding to one image candidate area;

obtaining an enhanced semantic information set corresponding to the regional semantic information set using a Graph Convolutional Network (GCN), wherein each enhanced semantic information of the enhanced semantic information set is one of the regional semantic information sets. corresponding to the region semantic information of , wherein the GCN is configured to establish an association relationship between the various region semantic information;

obtaining a degree of matching between the respective enhanced semantic information and a set of text features corresponding to the text to be found using an image region finding network model, wherein the image region finding network model is configured between the image candidate region and the text to be found. determine a matching relationship of , wherein each word of the text to be found corresponds to a single word feature of the set of text features; and

and determining a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each of the enhanced semantic information.

According to a second aspect of the present disclosure, there is provided a model training method,

obtaining a set of text to be trained and a set of image candidate regions to be trained, wherein the set of text to be trained includes a first text to be trained and a second text to be trained, and the set of image candidate regions to be trained is a first image to be trained image. a candidate region and a second candidate image to be trained region, wherein the first to-be-trained text and the first to-be-trained image candidate region have a matching relationship, and the first trained text and the second to-be-trained image candidate region does not have a matching relationship, the second to-be-trained text and the second to-be-trained image candidate region have a matching relationship, and the second to-be-trained text and the first to-be-trained image candidate region have a matching relationship. do not have ―;

determining a target loss function according to the first to-be-trained text, the second to-be-trained text, the first to-be-trained image candidate region, and the second to-be-trained image candidate region; and

training an image region finding network model to be trained using the target loss function to obtain an image region finding network model, wherein the image region finding network model is to be found with image candidate regions according to a text feature set and enhanced semantic information. and determine a matching relationship between text, wherein the enhanced semantic information and the image candidate region have a correspondence, and the text feature set and the text to be found have a correspondence.

According to a third aspect of the present disclosure, there is provided an apparatus for finding an image region,

a generating module, configured to generate a set of region semantic information according to a set of image candidate regions of an image to be found, wherein each region semantic information of the set of region semantic information corresponds to one image candidate region of the set of image candidate regions;

an acquiring module, configured to acquire, using GCN, an enhanced semantic information set corresponding to the regional semantic information set generated by the generating module, wherein each enhanced semantic information of the enhanced semantic information set is one of the regional semantic information set Corresponding to the domain semantic information of , the GCN is configured to establish an association between the various domain semantic information,

The obtaining module is further configured to obtain a matching degree between the text feature set corresponding to the text to be found using the image area finding network model and the respective enhanced semantic information, wherein the image area finding network model is configured to: determine a matching relationship between a candidate region and the text to be found, wherein each word of the text to be found corresponds to one word feature of the set of text features; and

and a determining module, configured to determine a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each enhanced semantic information obtained by the acquiring module.

According to a fourth aspect of the present disclosure, there is provided a model training apparatus,

an acquiring module configured to acquire a set of text to be trained and a set of image candidate regions to be trained, wherein the set of text to be trained includes a first text to be trained and a second text to be trained, and wherein the set of image candidate regions to be trained is a first training an image candidate region to be trained and a second image candidate region to be trained, wherein the first to-be-trained text and the first to-be-trained image candidate region have a matching relationship, and the first trained text and the second to-be-trained region The image candidate region does not have a matching relationship, the second to-be-trained text and the second to-be-trained image candidate region have a matching relationship, and the second to-be-trained text and the first to-be-trained image candidate region match not in a relationship ―;

a determining module, configured to determine a target loss function according to the first to-be-trained text, the second to-be-trained text, the first to-be-trained image candidate region, and the second to-be-trained image candidate region obtained by the acquiring module ; and

a training module, configured to train an image region finding network model to be trained using the target loss function determined by the determining module to obtain an image region finding network model, wherein the image region finding network model includes a text feature set and enhanced semantics and determine a matching relationship between the image candidate region and the text to be found according to the information, wherein the enhanced semantic information and the image candidate region have a correspondence, and the text feature set and the text to be found have a correspondence. - includes

According to a fifth aspect of the present disclosure, there is provided a terminal device,

including memory, transceiver, processor and bus systems;

The memory is configured to store a program,

The processor is

generating a set of region semantic information according to a set of image candidate regions of an image to be found, wherein each region semantic information of the set of region semantic information corresponds to one image candidate region of the set of image candidate regions;

obtaining an enhanced semantic information set corresponding to the regional semantic information set using a GCN, wherein each enhanced semantic information in the enhanced semantic information set corresponds to one region semantic information of the regional semantic information set, wherein the GCN is configured to establish associations between various domain semantic information;

obtaining a degree of matching between the respective enhanced semantic information and a set of text features corresponding to the text to be found using an image region finding network model, wherein the image region finding network model is between the image candidate region and the text to be found. determine a matching relationship of , wherein each word of the text to be found corresponds to a single word feature of the set of text features; and

and execute a program in the memory to perform an operation of determining a target image candidate region from the set of image candidate regions according to a degree of matching between the text feature set and each of the enhanced semantic information;

The bus system is configured to connect the memory and the processor so that the memory and the processor can communicate.

In possible designs, in possible implementations of the fifth aspect of embodiments of the present disclosure, the processor comprises:

an operation of obtaining region semantic information corresponding to each image candidate region using a convolutional neural network (CNN), wherein the image candidate region includes region information, the region information comprising a location of an image candidate region of the image to be found information and size information of the image candidate region; and

An operation of generating the region semantic information set according to the N regions semantic information when region semantic information corresponding to N image candidate regions is obtained, wherein N is an integer greater than or equal to 1

and further configured to execute a program in the memory to perform

In possible designs, in possible implementations of the fifth aspect of embodiments of the present disclosure, the processor comprises:

obtaining first region semantic information and second region semantic information from the region semantic information set, wherein the first region semantic information is any one region semantic information of the region semantic information set, and the second region semantic information is any one region semantic information of the region semantic information set;

obtaining a strength of a connection edge between the first region semantic information and the second region semantic information;

normalizing the strength of a connecting edge between the first region semantic information and the second region semantic information to obtain a normalized strength;

determining a target connection matrix according to normalized strengths between various region semantic information of the region semantic information set; and

Determining an enhanced semantic information set corresponding to the target connection matrix using the GCN

and further configured to execute a program in the memory to perform

In possible designs, in possible implementations of the fifth aspect of embodiments of the present disclosure, the processor comprises:

generating a connection matrix according to normalized strengths between various region semantic information of the region semantic information set; and

generating the target connection matrix according to the connection matrix and the unit matrix

and further configured to execute a program in the memory to perform

In possible designs, in possible implementations of the fifth aspect of embodiments of the present disclosure, the processor comprises:

the following way

further configured to execute the program in the memory to perform the operation of calculating the enhanced semantic information with

는 GCN의 k 번째 계층에 대응하는 i 번째 향상된 시맨틱 정보를 나타내고,

는 GCN의 k-1 번째 계층에 대응하는 j 번째 향상된 시맨틱 정보를 나타내며,

는 GCN의 k 번째 계층의 제1 네트워크 파라미터를 나타내고,

는 GCN의 k 번째 계층의 제2 네트워크 파라미터를 나타내며,

는 j 번째 노드가 i 번째 노드의 이웃 노드임을 나타내고,

는 상기 타깃 연결 매트릭스의 요소를 나타낸다.here

represents the i-th enhanced semantic information corresponding to the k-th layer of GCN,

represents the j-th enhanced semantic information corresponding to the k-1 th layer of GCN,

represents the first network parameter of the k-th layer of GCN,

represents the second network parameter of the k-th layer of GCN,

indicates that the j-th node is a neighbor of the i-th node,

denotes an element of the target connection matrix.

In possible designs, in possible implementations of the fifth aspect of embodiments of the present disclosure, the processor comprises:

obtaining the text to be found;

obtaining a text vector sequence according to the text to be found, the text vector sequence comprising T word vectors, each word vector corresponding to one word, and T being an integer greater than or equal to one;

encoding each word vector of the sequence of text vectors to obtain a text feature; and

generating the text feature set according to the T text features when text features corresponding to the T word vectors are obtained

and execute a program in the memory to perform

In possible implementations of the fifth aspect of the embodiments of the present disclosure, the processor comprises:

the following way

and execute a program in the memory to perform an operation of obtaining the text feature with

는 상기 텍스트 특징 세트의 t 번째 텍스트 특징을 나타내고,

는 장단기 메모리(Long Short-Term Memory, LSTM) 네트워크를 사용하여 인코딩을 수행하는 것을 나타내며,

는 상기 텍스트 벡터 시퀀스의 t 번째 단어 벡터를 나타내고,

는 상기 텍스트 특징 세트의 (t-1) 번째 텍스트 특징을 나타낸다.here

denotes the t-th text feature of the set of text features,

indicates that encoding is performed using a Long Short-Term Memory (LSTM) network,

represents the t-th word vector of the text vector sequence,

denotes the (t-1)-th text feature of the text feature set.

According to a sixth aspect of the present disclosure, a server is provided,

including memory, transceiver, processor and bus systems;

The memory is configured to store a program,

The processor is

generating a set of region semantic information according to a set of image candidate regions of an image to be found, wherein each region semantic information of the set of region semantic information corresponds to one image candidate region of the set of image candidate regions;

obtaining an enhanced semantic information set corresponding to the regional semantic information set using a GCN, wherein each enhanced semantic information in the enhanced semantic information set corresponds to one region semantic information of the regional semantic information set, wherein the GCN is configured to establish associations between various domain semantic information;

obtaining a degree of matching between the respective enhanced semantic information and a set of text features corresponding to the text to be found using an image region finding network model, wherein the image region finding network model is between the image candidate region and the text to be found. determine a matching relationship of , wherein each word of the text to be found corresponds to a single word feature of the set of text features; and

determining a target image candidate region from the image candidate region set according to a degree of matching between the text feature set and each of the enhanced semantic information;

configured to execute a program in the memory to perform

The bus system is configured to connect the memory and the processor so that the memory and the processor can communicate.

According to a seventh aspect of the present disclosure, a server is provided,

including memory, transceiver, processor and bus systems;

The memory is configured to store a program,

The processor is

obtaining a set of text to be trained and a set of image candidate regions to be trained, wherein the set of text to be trained includes a first text to be trained and a second text to be trained, and wherein the set of image candidate regions to be trained is a first image to be trained image. a candidate region and a second candidate image to be trained region, wherein the first to-be-trained text and the first to-be-trained image candidate region have a matching relationship, and the first trained text and the second to-be-trained image candidate region does not have a matching relationship, the second to-be-trained text and the second to-be-trained image candidate region have a matching relationship, and the second to-be-trained text and the first to-be-trained image candidate region have a matching relationship. do not have ―;

determining a target loss function according to the first to-be-trained text, the second to-be-trained text, the first to-be-trained image candidate region, and the second to-be-trained image candidate region; and

training an image region finding network model to be trained using the target loss function to obtain an image region finding network model, wherein the image region finding network model is to be found with image candidate regions according to a text feature set and enhanced semantic information. determine a matching relationship between text, wherein the enhanced semantic information and the image candidate region have a correspondence, and the text feature set and the text to be found have a correspondence;

configured to execute a program in the memory to perform

The bus system is configured to connect the memory and the processor so that the memory and the processor can communicate.

In a possible design, in a first implementation of the seventh aspect of an embodiment of the present disclosure, the processor comprises:

the following way

and execute the program in the memory to perform an operation of determining the target loss function with

은 상기 타깃 손실 함수를 나타내고,

는 상기 제1 훈련될 이미지 후보 영역을 나타내며,

는 상기 제1 훈련될 텍스트를 나타내고,

는 상기 제2 훈련될 이미지 후보 영역을 나타내며,

는 상기 제2 훈련될 텍스트를 나타내고,

는 훈련될 데이터 쌍을 나타내며, max()는 최대값을 취함을 나타내고,

은 제1 파라미터 제어 가중치를 나타내며,

는 제2 파라미터 제어 가중치를 나타내고,

은 제1 미리 설정된 임계값을 나타내며,

는 제2 미리 설정된 임계값을 나타낸다.here

represents the target loss function,

represents the first to-be-trained image candidate region,

represents the first to-be-trained text,

represents the second to-be-trained image candidate region,

represents the second to-be-trained text,

denotes the data pair to be trained, max() denotes to take the maximum,

represents the first parameter control weight,

represents the second parameter control weight,

represents the first preset threshold,

denotes a second preset threshold value.

According to an eighth aspect of the present disclosure, there is provided a computer-readable storage medium, the computer-readable storage medium storing instructions, the instructions, when executed in a computer, causing the computer to display in any one of the aforementioned aspects. follow the method to be followed.

According to a ninth aspect of the present disclosure, there is provided a method for finding an image region,

receiving an image finding command;

obtaining an image candidate region set of an image to be found according to the image finding command in response to the image finding command, wherein the image candidate region set includes N image candidate regions, where N is an integer greater than or equal to one; ;

generating a set of region semantic information according to the set of image candidate regions, wherein the set of region semantic information includes N pieces of region semantic information, each region semantic information corresponding to one image candidate region;

obtaining an enhanced semantic information set corresponding to the region semantic information set using GCN, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information corresponding to one region semantic information, and , the GCN is configured to establish associations between various domain semantic information;

obtaining a set of text features corresponding to the text to be looked up, wherein the text to be found comprises T words, the set of text features comprises T word features, each word corresponding to one word feature, and , T is an integer greater than or equal to 1 —;

obtaining a degree of matching between the set of text features and each of the enhanced semantic information using an image region finding network model, wherein the image region finding network model determines a matching relationship between the image candidate region and the text to be found configured to ―;

determining a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each of the enhanced semantic information; and

sending the image creation command to the client so that the client can display the target image candidate region according to the image creation command

includes

1 is a schematic architectural diagram of a system for finding an image region according to an embodiment of the present disclosure;
2 is a schematic diagram of an overall framework for finding an image region according to an embodiment of the present disclosure;
3 is a schematic diagram of an embodiment of a method for finding an image region according to an embodiment of the present disclosure;
4 is a schematic diagram of an embodiment of a model training method according to an embodiment of the present disclosure;
5 is a schematic diagram of an embodiment of an apparatus for finding an image region according to an embodiment of the present disclosure;
6 is a schematic diagram of an embodiment of a model training apparatus according to an embodiment of the present disclosure;
7 is a schematic structural diagram of a terminal device according to an embodiment of the present disclosure.
8 is a schematic structural diagram of a server according to an embodiment of the present disclosure.

In the related art, the image region that best matches the natural sentence can be found in the image, but the spatial relationship between the local regions is not considered, and the semantic information between the local regions is ignored. As a result, the target image region cannot be accurately found, which deteriorates the image understanding ability.

An embodiment of the present disclosure provides a method for finding an image region, a model training method, and a related apparatus. The semantic representation between image candidate regions can be effectively enhanced using GCN, and spatial relationships between image candidate regions are considered, which can improve the accuracy of finding image regions, thereby improving image comprehension ability.

In the specification and claims of the present invention and the accompanying drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects, and It is not necessarily used to describe a sequence or sequence. Data used in this manner may be interchanged in appropriate circumstances such that the embodiments of the present disclosure described herein may be implemented in orders other than those shown or described herein. Also, the terms “comprises,” “corresponds to,” and any other variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to the explicitly listed steps or units, but is not explicitly listed or includes such process, method, product or apparatus. may include other steps or units that are not unique to

The method for finding an image region provided in the present disclosure is applicable to fields such as image processing and mode recognition to find a target of interest in an image, so that a specific type of target can be determined and a bounding box of the target can be provided. Methods for finding image regions are widely applied in fields such as face recognition, medical imaging, intelligent video monitoring, robot navigation, content-based image search, image-based drawing technology, image editing, and augmented reality. For example, in the scenario of content-based image search, suppose there is image A. A plurality of candidate regions are extracted from image A. It is assumed that the user has entered the sentence "The boy is holding an apple". In this case, the sentence is matched with each candidate area. A target candidate area is selected from among a plurality of candidate areas according to the matching result. The present disclosure is mainly for completing natural sentence image finding using GCN. The natural sentence may be a word, a phrase, or a sentence, and a target candidate region of an image corresponding to the natural sentence is found. The target candidate area may be defined as a rectangular box.

During practical application, image region finding may involve three levels. The first level is the image level. That is, it is determined whether there is an associated target object in the image. In image classification or image annotation techniques, for example, for the word "apple", an "apple" object may be circled in the image.

The second layer is the domain level. That is, it is determined whether the region of the image contains the target type. For example, in detection of a target type in an image, for example for "a boy is holding an apple", a region may be framed from the image. This area includes boys and apples.

The third layer is the domain level. That is, the type of target object to which each pixel of the image belongs is determined. Pixel level segmentation also includes type level target segmentation and semantic segmentation. The main difference between type level target segmentation and semantic segmentation is that in semantic segmentation, all targets including the background of the image have to be segmented, and the type of target has to be determined, whereas the target of interest only needs to be segmented and the target classified. .

For convenience of description, the present disclosure proposes a method for finding an image region. This method is applicable to the system for finding an image region shown in FIG. 1 . 1 is a schematic architectural diagram of a system for finding an image region according to an embodiment of the present disclosure; As shown in the figure, the method for finding an image region provided in the present disclosure may be applied to a server or a client. When this method is applied to the server, after determining the finding result, the server may send the finding result to the client, and display the corresponding target image candidate area using the client. When this method is applied to the client, after determining the finding result, the client may directly display the corresponding target image candidate area. Specifically, for one image, an image detection method is first used to obtain a plurality of image candidate regions (ie, local regions of the image). For a plurality of image candidate regions, the spatial relationship between the image candidate regions is used to build a graph. Then, for image candidate regions, CNN can be used to extract the corresponding semantic features. Based on the obtained semantic features and the constructed graph, the GCN is used to further learn the representation of the image candidate region. Based on the representation of image candidate regions obtained using GCN, the semantic relevance between these image candidate regions and a given natural sentence is calculated to further determine the most correlated image candidate regions as the final result of natural sentence image finding. A semantic matching method is used to measure.

The client is placed in the terminal device. Terminal devices include, but are not limited to, tablet computers, notebook computers, palmtop computers, mobile phones, voice interaction devices, and personal computers (PCs). Voice interaction devices include, but are not limited to, smart sound and smart appliances.

The method for finding an image region proposed in the present disclosure may provide a natural sentence image finding service. This service may be located on the server side or may be located on the terminal device side. The application of the method for finding image regions on the server side can complete a deeper understanding of the image to further perform fine-grained annotations on the image, thus helping users to make fast and precise search and matching. It can be appreciated that it can also be applied to personal recommendations of picture and text information. The method for finding the image area may be arranged in a terminal device, for example, a mobile phone terminal or a robot. The robot's camera acquires a corresponding image signal, and the user interacts with the corresponding robot using natural language. For example, after the user obtains the corresponding natural language text through voice or keyboard input, the image region finding network model may be used to find the local region of the image for the corresponding natural language text. In this way, the terminal device can better interact with the user.

In an example scenario, a user may conveniently perform an accurate search. A user may input natural language text into the terminal device using a voice or a keyboard. The terminal device uses the method for finding an image region in an embodiment of the present disclosure to determine a region having a degree of best matching with the natural language text in the image to be found. This has practical implications in the field of criminal investigation and education. For example, in the field of criminal investigation, it is possible to accurately find a characteristic suspect in a surveillance video image. Alternatively, in the field of education, all students can be accurately found in the class video image. No complicated manual screening is required, users only need to input natural language text.

In an exemplary scenario, the service terminal may conveniently perform a personalized recommendation for the terminal device corresponding to the user. The service terminal collects the fully authorized natural language text entered by the user, and the service terminal sees this to push similar image resources, video resources, web page resources, etc. to the selected area so that more accurate personalized recommendations can be achieved. In the embodiment of the disclosure, by using the method for finding an image region to determine a region having a degree of best matching with a natural language in the image to be found, the accuracy of the resource recommendation process may be improved.

이 추가로 획득되며, 여기서 n은 이미지로부터 추출된 이미지 후보 영역의 총 개수를 나타낸다. 그 후, GCN은 이미지 후보 영역의 향상된 시맨틱 표현

을 획득하기 위해 이미지 후보 영역의 추출된 시맨틱 표현을 향상시키는 데 사용된다. GCN을 구축하는 과정에서, 대응하는 그래프를 구축하고 대응하는 연결 에지 정보를 추가로 정의하기 위해 이미지 후보 영역 사이의 시맨틱 유사성이 고려된다. 연결 에지 정보는 대응하는 이미지 후보 영역의 시맨틱 표현을 향상시키는 데 사용된다.For convenience of understanding, referring to FIG. 2 , FIG. 2 is a schematic diagram of an overall framework for finding an image region according to an embodiment of the present disclosure. As shown in the figure, in the case of a natural image, the object suggestion method is used to obtain a corresponding image candidate region. After the corresponding image candidate region is extracted, the corresponding CNN is used to extract the semantic representation of the corresponding image candidate region, and each candidate region is represented by one feature vector. Semantic representations corresponding to corresponding candidate regions

is further obtained, where n represents the total number of image candidate regions extracted from the image. After that, the GCN is an enhanced semantic representation of the image candidate region.

is used to enhance the extracted semantic representation of the image candidate region to obtain In the process of constructing the GCN, the semantic similarity between image candidate regions is considered in order to construct a corresponding graph and further define the corresponding connected edge information. The connecting edge information is used to enhance the semantic representation of the corresponding image candidate region.

For an input natural sentence (eg, "white man playing baseball on the left"), the RNN is used to encode the natural sentence to obtain a semantic representation corresponding to the natural sentence. For the semantic representation of the natural sentence and the corresponding enhanced semantic representation of the image candidate region, a matching learning method is used to learn the semantic relationship between the natural sentence and the corresponding image candidate region. Finally, the semantic similarity between natural sentences and image candidate regions is used to select the most semantically relevant image candidate regions as target image candidate regions.

Referring to the foregoing description, a method for finding an image region in the present disclosure is described below. Referring to FIG. 3 , an example in which the method is applied to an apparatus for finding an image region is used for explanation. The device for finding the image region may be disposed in a server or may be disposed in a terminal device. An embodiment of a method for finding an image region in an embodiment of the present disclosure includes the following steps.

Step 101. The apparatus for finding an image region obtains a set of image candidate regions of an image to be found, the image candidate region set includes N image candidate regions, where N is an integer greater than or equal to one.

In this embodiment, the apparatus for finding an image region first acquires an image to be found. The image to be found may be an image stored on the backend server, or it may be an image uploaded by the client, or it may be a local image of the client. It may be understood that the device for finding the image area may be disposed in a server or may be disposed in a terminal device. It is not limited here.

After the apparatus for finding an image region obtains an image to be found, an object suggestion method may be used to extract a set of image candidate regions from the image to be found. The image candidate region set includes N image candidate regions, where N is an integer greater than or equal to 1. When N is equal to 1, it indicates that there is only one image candidate region in the image to be found, and the image candidate region is directly used as the target image candidate region.

Based on the object suggestion method, an image candidate region in the image may be extracted. Specifically, an image candidate area, which is a location where a target may appear in the image, is found in advance. Using information such as texture, edge, and color of the image ensures that a relatively high intersection-over-union (IoU) can be maintained while a relatively small number of windows are selected. Object proposal methods include, but are not limited to, region-based CNN (R-CNN), fast R-CNN, and faster R-CNN, and the like. It is not limited here.

Step 102. The apparatus for finding an image region generates a set of region semantic information according to the image candidate region set, the region semantic information set includes N regions semantic information, and each region semantic information is in one image candidate region. respond

In this embodiment, after obtaining the image candidate region set, the apparatus for finding the image region generates a corresponding semantic representation of the image candidate region by using the neural network to obtain the region semantic information set. The region semantic information set includes N pieces of region semantic information. Each region semantic information corresponds to one image candidate region.

The neural network may specifically be a CNN. In practical application, the neural network may be another type of neural network. This is only schematic here and should not be construed as a limitation on the present disclosure.

In the above process, that is, the apparatus for finding an image region generates a set of region semantic information according to a set of image candidate regions of an image to be found, and each region semantic information of the set of region semantic information is one of the set of image candidate regions. Corresponds to the image candidate area.

Step 103. The apparatus for finding an image region obtains an enhanced semantic information set corresponding to the region semantic information set by using the GCN, wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information is one Corresponding to the domain semantic information of , the GCN is configured to establish an association relationship between the various domain semantic information.

That is, each enhanced semantic information of the enhanced semantic information set corresponds to one region semantic information of the region semantic information set.

In this embodiment, the apparatus for finding an image region uses GCN to obtain an enhanced semantic information set corresponding to the region semantic information set. That is, the semantic representation of image candidate regions can be improved using GCN. The enhanced semantic information set includes N enhanced semantic information. That is, each image candidate region corresponds to one region semantic information, and each image candidate region corresponds to one region semantic enhancement information. GCN can be used to establish associations between nodes. In the present disclosure, an association relationship may be established between various region semantic information.

GCN is a convolutional network model. Corresponding to the GCN, the goal of the GCN is to learn the mapping of signals or features in the graph G = (V, E). The process of constructing the graph is performed after the image candidate area is obtained. A graph is built according to the spatial information between image candidate regions. The information contained in the data and the relationship between the data can be used to improve the semantic representation of the image candidate region so that enhanced semantic information can be obtained.

Step 104. The apparatus for finding the image region obtains a set of text features corresponding to the text to be found, the text to be found includes T words, the text feature set includes T word features, and each word includes: Corresponds to one word feature, and T is an integer greater than or equal to 1.

In the present embodiment, the apparatus for finding an image region acquires a text to be found. It may be understood that step 104 may be performed before step 101 , may be performed after step 103 , or may be performed concurrently with step 101 . The execution order of step 104 is not limited here. The text to be searched for may specifically be a text input by the user, or may be a text obtained by recognizing a voice input by the user. The text to be searched for is expressed in the form of words, phrases, sentences, paragraphs, and the like. The text to be searched for may be Chinese, English, Japanese, French, German, Russian, or the like. It is not limited here.

After the text to be found is obtained, feature extraction and encoding are performed for each word of the text to be found to finally obtain a set of text characteristics. For example, the text to be found contains four words "the boy is holding an apple", that is, the four words "boy", "holding", "one" and "apple". The features of the four words are extracted and then encoded to further obtain a set of text features. The text to be found contains T words. The text feature set includes T word features. Each word corresponds to one word feature. T is an integer greater than or equal to 1.

It may be understood that the text to be found is a concrete expression of "the boy is holding an apple". Accordingly, an image candidate area including both "boy" and "apple" can be obtained from the image to be found.

Step 105. The apparatus for finding an image region uses an image region finding network model to obtain a degree of matching between the text feature set and each enhanced semantic information, and the image region finding network model is configured to determine the distance between the image candidate region and the text to be found and determine a matching relationship.

In this embodiment, the apparatus for finding an image region may individually input each set of enhanced semantic information and text feature into an image region finding network model, and the image region finding network model outputs a corresponding matching degree. The image region finding network model is configured to determine a matching relationship between the image candidate region and the text to be found. That is, when the matching degree is high, it indicates that the matching relationship is strong.

It will be appreciated that the degree of matching may be expressed as a match score or match identifier, or may be expressed as another type of match relationship.

In the above process, that is, the apparatus for finding an image region uses the image region finding network model to obtain a matching degree between a set of text features corresponding to the text to be found and each enhanced semantic information. Each word of the text to be found corresponds to one word feature of the text feature set.

Step 106. The apparatus for finding an image region determines a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each enhanced semantic information.

In the present embodiment, the apparatus for finding an image region may select an image candidate region having the highest matching degree among image candidate regions as the target image candidate region according to the matching degree between the text feature set and each enhanced semantic information. For convenience of explanation, [Table 1] is a schematic degree of matching between the text feature set and the enhanced semantic information.

As can be seen from [Table 1], the matching degree corresponding to "text feature set + enhanced semantic information D" is maximum. Accordingly, the apparatus for finding the image region uses the image candidate region D as the target image candidate region.

In an embodiment of the present disclosure, a method for finding an image region is provided. The method includes first obtaining a set of image candidate regions in an image to be found, the set of image candidate regions comprising N image candidate regions, generating a set of region semantic information according to the next set of image candidate regions, each The domain semantic information of ?汰? Corresponding to an image candidate region—obtaining an enhanced semantic information set corresponding to the region semantic information set using the next GCN—each enhanced semantic information corresponds to one region semantic information, and the GCN is between various region semantic information. configured to build an association relationship of -, further, obtaining a set of text features corresponding to the text to be found, then using an image region finding network model to determine the degree of matching between the set of text features and each enhanced semantic information obtaining, and finally determining a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each enhanced semantic information. In the above-described manner, the semantic representation between image candidate regions can be effectively improved by using GCN, and spatial relationships between image candidate regions are considered, so that the accuracy of finding image regions can be increased, thereby improving image understanding ability. can

Optionally, based on the above-described embodiment corresponding to FIG. 3 , in a first optional embodiment of the method for finding an image region provided in an embodiment of the present disclosure, by an apparatus for finding an image region, an image candidate region The step of generating a set of region semantic information according to the set comprises:

obtaining, by the apparatus for finding an image region, region semantic information corresponding to each image candidate region by using CNN, wherein the image candidate region includes region information, and the region information includes information about the image candidate region in the image to be found. including location information and size information of the image candidate area; and

generating, by the apparatus for finding an image region, a set of region semantic information according to the N region semantic information when region semantic information corresponding to the N image candidate regions is obtained;

may include.

로서 정의된다고 가정한다. 각각의 이미지 후보 영역은 영역 정보

를 포함한다. 이미지 후보 영역

는 이미지 후보 영역 세트의 하나의 이미지 후보 영역을 나타낸다.

및

은 찾아질 이미지에서 이미지 후보 영역의 위치 정보를 나타낸다. 구체적으로,

는 찾아질 이미지에서 이미지 후보 영역의 가장 높은 지점의 수평 좌표 위치 정보를 나타내고,

은 찾아질 이미지에서 이미지 후보 영역의 가장 높은 지점의 수직 좌표 위치 정보를 나타내며,

및

은 이미지 후보 영역의 크기 정보를 나타내고, 크기 정보는 찾아질 이미지에 대한 이미지 후보 영역의 비례 크기이며,

은 찾아질 이미지에서 이미지 후보 영역의 폭 정보를 나타내고,

은 찾아질 이미지에서 이미지 후보 영역의 높이 정보를 나타낸다.In this embodiment, after obtaining a set of image candidate regions, the apparatus for finding an image region may generate region semantic information of each image candidate region using CNN, wherein the region semantic information is the semantic representation of the image candidate region. to be. Specifically, the set of image candidate regions is

Assume that it is defined as Each image candidate area contains area information

includes image candidate area

denotes one image candidate region of the image candidate region set.

and

denotes position information of an image candidate region in the image to be found. Specifically,

represents the horizontal coordinate position information of the highest point of the image candidate area in the image to be found,

represents the vertical coordinate position information of the highest point of the image candidate area in the image to be found,

and

represents the size information of the image candidate area, the size information is the proportional size of the image candidate area with respect to the image to be found,

represents the width information of the image candidate region in the image to be found,

denotes height information of the image candidate region in the image to be found.

은 다음의 표현image candidate area

is the expression

is input to CNN to obtain

가 획득된다. 전술한 방식에서, 이미지 후보 영역 세트

에 대응하는 영역 시맨틱 정보 세트

가 획득되며, 여기서 n은 1보다 크거나 같고 N보다 작거나 같은 정수이다.In this way, the corresponding region semantic information

is obtained In the above manner, the image candidate region set

A set of region semantic information corresponding to

is obtained, where n is an integer greater than or equal to 1 and less than or equal to N.

For convenience of understanding, CNN generally includes several layers, namely a convolutional layer, a rectified linear unit (ReLU) layer, a pooling layer, and a fully connected layer.

In the case of convolutional layers, each convolutional layer in CNN is formed by a plurality of convolutional units. The parameters of each convolution unit are obtained through optimization using a backpropagation algorithm. The purpose of the convolution operation is to extract the input different features. The first convolutional layer may extract only low-level features, for example, levels such as edges, lines, angles, etc., and the network with more layers may iteratively extract more complex features from the low-level features.

In the case of the ReLU layer, Linear Rectification (ReLU) is used for the activation function in the neural network layer.

In the case of a pooling layer, in general, after a convolution layer, a feature of a very large dimension is obtained, the feature is divided into a plurality of regions, and the maximum or average value of a new feature of a relatively small dimension is obtained. taken for

For a fully-connected layer, all local features are combined to form a global feature and used to calculate the final score of each type.

Next, in an embodiment of the present disclosure, a method for generating a region semantic information set is provided. First, region semantic information corresponding to the image candidate region is obtained using CNN. The image candidate region includes region information. The region information includes position information of the image candidate region and size information of the image candidate region in the image to be found. The region semantic information set is generated according to the N region semantic information when region semantic information corresponding to the N image candidate regions is obtained. In the above-described manner, region semantic information of each image candidate region may be extracted using CNN. CNN is a feed-forward neural network. The artificial neurons of CNNs can respond to neighboring units in the partial coverage area, so they have excellent performance for large-scale image processing. In this way, the accuracy of information extraction is improved.

Optionally, based on the above-described embodiment corresponding to FIG. 3 , in a second optional embodiment of the method for finding an image region provided in an embodiment of the present disclosure, by the apparatus for finding an image region, using GCN to obtain an improved semantic information set corresponding to the region semantic information,

obtaining, by the apparatus for finding an image region, first region semantic information and second region semantic information from the region semantic information set, wherein the first region semantic information is any one region semantic information in the region semantic information set; the second region semantic information is any one region semantic information in the region semantic information set;

obtaining, by the apparatus for finding an image region, a strength of a connecting edge between the first region semantic information and the second region semantic information;

normalizing, by the image region value, the strength of a connecting edge between the first region semantic information and the second region semantic information to obtain a normalized intensity;

determining, by the apparatus for finding an image region, a target connection matrix according to normalized strengths between various region semantic information of a region semantic information set; and

determining, by the apparatus for finding an image region, an enhanced semantic information set corresponding to a target connection matrix using the GCN;

may include.

In this embodiment, the apparatus for finding an image region uses one GCN to complete the enhancement of the semantic representation of the image candidate region. First, one graph needs to be built. In this graph, each node corresponds to region semantic information of the image candidate region. Between every node there is a corresponding connecting edge. The strength of the connection information of the connection edge comes from one deep network for prediction.

는 제1 영역 시맨틱 정보를 나타내고,

는 제2 영역 시맨틱 정보를 나타내며,

는 심층망을 나타내고, 구체적으로 다층 퍼셉트론(multi-layer perceptron), 벡터 내적 또는 코사인 유사성을 사용하여 구현될 수 있다.

는 제1 영역 시맨틱 정보와 제2 영역 시맨틱 정보 사이의 연결 에지의 강도를 나타낸다. 다음으로,

는 정규화된 강도를 획득하기 위해 정규화된다. 그 다음, 영역 시맨틱 정보 세트 내의 다양한 영역 시맨틱 정보 사이의 정규화된 강도에 따라 타깃 연결 매트릭스가 결정된다. 마지막으로, GCN을 사용하여 타깃 연결 매트릭스에 대응하는 향상된 시맨틱 정보 세트가 생성된다.here,

represents the first region semantic information,

represents the second area semantic information,

represents a deep network, and specifically can be implemented using a multi-layer perceptron, vector dot product, or cosine similarity.

denotes the strength of a connection edge between the first region semantic information and the second region semantic information. to the next,

is normalized to obtain a normalized intensity. Then, a target connection matrix is determined according to the normalized strength between the various region semantic information in the region semantic information set. Finally, a set of enhanced semantic information corresponding to the target connectivity matrix is generated using GCN.

A graph is a data format. The graph can be used to represent social networks, communication networks, protein molecular networks, and the like. Nodes in the graph represent individuals in the network. A connection edge represents a connection relationship between individuals. Many machine learning tasks require the use of graph-structured data. Therefore, the emergence of GCN provides a new concept to solve these problems. A convolutional slice can be built using three steps. Step 1, selecting a node sequence of a fixed length from the graph; Step 2, collecting a fixed-size set of neighboring domains for each node of the sequence; and Step 3, normalizing the subgraph formed by the current node and the neighboring domain corresponding to the node for use as input to the convolutional structure. After all the convolutional slices are built using the three steps described above, the convolution structure is used to perform operations on each slice individually.

Next, in an embodiment of the present disclosure, a method of obtaining an enhanced semantic information set using GCN is provided. First, the first region semantic information and the second region semantic information are obtained from the region semantic information set. Then, the strength of the connecting edge between the first region semantic information and the second region semantic information is obtained. Next, the strength of the connecting edge between the first region semantic information and the second region semantic information is normalized to obtain a normalized strength. Then, a target connection matrix is determined according to the normalized strength between the various region semantic information in the region semantic information set. Finally, the set of enhanced semantic information corresponding to the target connectivity matrix is determined using the GCN. In the above-described manner, a semantic relationship between image candidate regions is established using GCN. In this way, the spatial information and the semantic relationship are sufficiently taken into account, thereby improving the image-based finding performance.

Optionally, based on the above-described second embodiment corresponding to FIG. 3 , in a third optional embodiment of the method for locating an image region provided in an embodiment of the present disclosure, by an apparatus for locating an image region, the region Determining the target connection matrix according to the normalized strength between the semantic information of various regions of the semantic information set comprises:

generating, by the apparatus for finding an image region, a connection matrix according to normalized strengths between various region semantic information in a region semantic information set; and

generating a target connection matrix according to the connection matrix and the unit matrix, by the device for finding the image region;

may include.

In the present embodiment, the apparatus for finding an image region may first normalize the intensity of a connecting edge between the first region semantic information and the second region semantic information to obtain a normalized intensity. Based on the above-described embodiment, the strength of the connecting edge between the first region semantic information and the second region semantic beauty information is specifically:

is expressed as

는 제1 영역 시맨틱 정보를 나타내고,

는 제2 영역 시맨틱 정보를 나타내며,

,

,

,

는 모두 GCN의 모델 파라미터이고,

는 제1 영역 시맨틱 정보와 제2 영역 시맨틱 정보 사이의 연결 에지의 강도를 나타낸다.here,

represents the first region semantic information,

represents the second area semantic information,

,

,

,

are all model parameters of GCN,

denotes the strength of a connection edge between the first region semantic information and the second region semantic information.

Normalized operation may be further performed on the corresponding edge information to obtain normalized strength,

는 제1 영역 시맨틱 정보와 제2 영역 시맨틱 정보 사이의 정규화된 강도를 나타낸다. 대응하는 영역 시맨틱 정보의 연결 정보가 완전한 연결 매트릭스를 추가로 구축하기 위해 모든 영역 시맨틱 정보가 트래버스(traverse)된다. here,

denotes the normalized strength between the first region semantic information and the second region semantic information. All the region semantic information is traversed in order to further construct the connection information of the corresponding region semantic information complete connection matrix.

To further enhance the information, the unit matrix is added to the corresponding complete connectivity matrix to obtain a target connectivity matrix.

Next, in an embodiment of the present disclosure, a method of determining a target connection matrix according to normalized strengths between various region semantic information in a region semantic information set is provided. That is, the connection matrix is first generated according to the normalized strength between the various region semantic information of the region semantic information set. Then, a target connection matrix is generated according to the connection matrix and the unit matrix. In the above-described manner, through the process measurement of normalization, the absolute value of a value in the physical system can be converted into a relative value relationship, so that the calculation can be simplified and the size can be reduced. Further, to further enhance the information, the unit matrix is added to the corresponding connection matrix to form a target connection matrix.

Optionally, based on the above-described second or third embodiment corresponding to FIG. 3 , in a fourth optional embodiment of the method for finding an image region provided in an embodiment of the present disclosure, in an apparatus for finding an image region By using the GCN, determining the enhanced semantic information set corresponding to the target connection matrix comprises:

calculating, by the device for finding an image region, an enhanced set of semantic information in the following manner,

는 GCN의 k 번째 계층에 대응하는 i 번째 향상된 시맨틱 정보를 나타내고,

는 GCN의 k-1 번째 계층에 대응하는 j 번째 향상된 시맨틱 정보를 나타내며,

는 GCN의 k 번째 계층의 제1 네트워크 파라미터를 나타내고,

는 GCN의 k 번째 계층의 제2 네트워크 파라미터를 나타내며,

는 j 번째 노드가 i 번째 노드의 이웃 노드임을 나타내고,

는 타깃 연결 매트릭스의 요소를 나타낸다. here,

represents the i-th enhanced semantic information corresponding to the k-th layer of GCN,

represents the j-th enhanced semantic information corresponding to the k-1 th layer of GCN,

represents the first network parameter of the k-th layer of GCN,

represents the second network parameter of the k-th layer of GCN,

indicates that the j-th node is a neighbor of the i-th node,

represents the elements of the target connection matrix.

In this embodiment, the step of improving the semantic representation of the image candidate region using GCN based on the target connection matrix by the apparatus for finding the image region may use the following equation,

는 GCN의 k 번째 계층의 제1 네트워크 파라미터를 나타내고,

는 GCN의 k 번째 계층의 제2 네트워크 파라미터를 나타낸다. GCN의 네트워크 파라미터는 그래프의 컨볼루션 계층 사이에 공유되지 않음이 이해될 수 있다. 그러나, 고정된 컨볼루션 계층에서, GCN의 네트워크 파라미터는 공유될 수 있거나 또는 공유되지 않을 수 있다. 노드 i에 대응하는 이웃 노드는 노드 J로서 선택된다. 노드 사이의 유사성은 노드 사이의 시맨틱 유사성을 사용하여 측정될 수 있다. 따라서, 하나의 완전히 연결된 그래프 구조가 구축되고, 이에 상응하여 각각의 노드는 다른 노드에 연결된다. 마지막으로, 각각의 노드의 대응하는 시맨틱 표현은 구축된 타깃 연결 매트릭스에 기초하여 업데이트된다.here,

represents the first network parameter of the k-th layer of GCN,

denotes the second network parameter of the k-th layer of GCN. It can be understood that the network parameters of the GCN are not shared among the convolutional layers of the graph. However, in a fixed convolutional layer, the network parameters of the GCN may or may not be shared. A neighboring node corresponding to node i is selected as node J. Similarity between nodes can be measured using semantic similarity between nodes. Thus, one fully connected graph structure is built, with each node correspondingly connected to another node. Finally, the corresponding semantic representation of each node is updated based on the built-up target connection matrix.

Multilayer graph convolution processing may be performed in GCN, and may be performed multiple times for the above-mentioned equations, and the same set of network parameters may be shared or network parameters may not be shared.

In addition, in an embodiment of the present disclosure, a specific manner of an enhanced semantic information set corresponding to a target connection matrix determined using GCN is provided. In the above-described manner, since a specific calculation method is provided for the GCN-based calculation, it is possible to improve the feasibility and operability of the solution.

Optionally, based on the above-described embodiment corresponding to FIG. 3 , in a fifth optional embodiment of the method for finding an image region provided in an embodiment of the present disclosure, by an apparatus for finding an image region, text to be found Obtaining a text feature set corresponding to

obtaining, by the apparatus for finding an image region, text to be found;

obtaining, by the apparatus for finding an image region, a text vector sequence according to the text to be found, wherein the text vector sequence includes T word vectors, each word vector corresponding to one word;

encoding, by the apparatus for finding an image region, each word vector in the text vector sequence to obtain a text feature; and

generating, by the apparatus for finding an image region, a text feature set according to the T text features when text features corresponding to the T word vectors are obtained;

may include.

In this embodiment, the apparatus for finding an image region first acquires a text to be found. The text to be found may be text input by the user, text converted from voice input by the user, or text extracted from the backend. After the text to be found is obtained, each word of the text to be found is extracted, and then a respective word vector is constructed for each word. It is assumed that the text to be found contains T words. In this case, T word vectors may be obtained. The T word vectors form a text vector sequence. A device for finding an image region encodes a sequence of text vectors using an LSTM network structure. Specifically, each word vector may be encoded using an LSTM structure to obtain T text features, thereby generating a set of text features.

In a natural language processing task, a method of representing a word in a computer is first considered. In general, there are two representations: a discrete representation (one-hot representation) and a distributed representation. In the one-hot representation, each word is represented by one long vector. The dimension of this vector is the word table size. In a vector, only one dimension has a value of 1 and the other dimensions are 0. These dimensions represent the current word. In this disclosure, the word vector dimension may be 300 dimensions. In word embeddings, words are transformed into distributed representations, or referred to as word vectors. There are several ways to create a word vector. These methods follow one idea that the meaning of any word can be expressed in words around it. A method of generating a word vector may include a statistical-based method and a language model-based method.

Next, in an embodiment of the present disclosure, a method for obtaining a text feature set is provided. That is, the text to be found is obtained first. Then, a text vector sequence is obtained according to the text to be searched for. The text vector sequence contains T word vectors. Each word vector corresponds to one word. Next, each word vector of the text vector sequence is encoded to obtain text features. A text dimensioning set is generated according to the T text features when text features corresponding to the T word vectors are obtained. In the above-described manner, the text to be found can be expressed in the form of features to further facilitate subsequent model prediction, thereby improving the feasibility and operability of the solution.

Optionally, based on the above-described fifth embodiment corresponding to FIG. 3 , in a sixth optional embodiment of the method for finding an image region provided in an embodiment of the present disclosure, by an apparatus for finding an image region, a text Encoding each word vector in the text vector sequence to obtain a feature comprises:

by the apparatus for finding an image region, it may include obtaining text features in the following manner,

는 텍스트 특징 세트에서 t 번째 텍스트 특징을 나타내고,

는 LSTM 네트워크를 사용하여 인코딩을 수행하는 것을 나타내며,

는 텍스트 벡터 시퀀스에서 t 번째 단어 벡터를 나타내고,

는 텍스트 특징 세트에서 (t-1) 번째 텍스트 특징을 나타낸다. here,

denotes the t-th text feature in the text feature set,

indicates that encoding is performed using the LSTM network,

denotes the t-th word vector in the text vector sequence,

denotes the (t-1)-th text feature in the text feature set.

에 대해, T는 찾아질 텍스트의 T개의 단어를 나타내고,

는 찾아질 텍스트에서 t 번째 단어를 나타낸다. 먼저, 찾아질 텍스트의 단어 벡터 표현은 각각의 단어의 단어 벡터 표현을 사용하여 획득될 수 있다. 즉, 텍스트 벡터 시퀀스

가 획득된다. 각각의 단어 벡터는 300개의 차원을 가질 수 있다. 찾아질 텍스트의 경우, LSTM 구조의 RNN이 찾아질 텍스트를 인코딩하는 데 사용된다. 즉, In this embodiment, the apparatus for finding an image region may encode each word vector using an LSTM structure to obtain a text feature. Entered text to be found

For , T denotes the T words of the text to be found,

denotes the t-th word in the text to be found. First, a word vector representation of the text to be found may be obtained using the word vector representation of each word. i.e. text vector sequence

is obtained Each word vector can have 300 dimensions. For the text to be searched for, the RNN of the LSTM structure is used to encode the text to be searched for. In other words,

가 획득되며, 여기서 LSTM 처리의 구체적인 방식은 다음과 같으며,In LSTM, the number of dimensions of the hidden state may be set to 512. The feature representation of the text to be found is obtained after processing. i.e. a set of text features

is obtained, where the specific manner of LSTM processing is as follows,

,

,

, 및

, and

이며

is

는 텍스트 벡터 시퀀스에서 t 번째 단어 벡터를 나타내고,

는 텍스트 특징 세트에서 (t-1) 번째 텍스트 특징을 나타내며,

는 입력 게이트를 나타내고,

는 포겟 게이트(forget gate)를 나타내며,

는 출력 게이트를 나타내고,

는 은닉 상태를 나타내며,

는 시그모이드 함수(sigmoid function)이고, tanh()는 쌍곡선 함수를 나타내며,

는 메모리 정보를 나타내고,

는 LSTM 파라미터를 나타내며,

는 점 곱셈(dot multiplication)을 나타내고,

는 변환 또는 매핑 매트릭스를 나타낸다.here,

denotes the t-th word vector in the text vector sequence,

denotes the (t-1)th text feature in the text feature set,

represents the input gate,

denotes a forget gate,

denotes the output gate,

represents the hidden state,

is a sigmoid function, tanh() denotes a hyperbolic function,

represents memory information,

represents the LSTM parameter,

represents dot multiplication,

denotes a transformation or mapping matrix.

LSTM is a long-term storage input. Certain units, called memory cells, are analogous to adders and gate control neurons, have weighted values at the next time step, and connect themselves to copy the actual values of their states and accumulated external signals. However, this self-connection is controlled by the multiplication gate to determine when to perform the learning of the other unit and erase the memory contents.

Next, in an embodiment of the present disclosure, a method of obtaining a text feature is provided. That is, the RNN of the LSTM structure is used to encode the word vector. In the above-described manner, if the network of the LSTM structure is used, the problem of vanishing gradient due to the gradual reduction of the backpropagation process can be solved. Specifically, in language processing tasks, LSTMs are suitable for handling temporal height related problems, such as machine translation, dialog generation, and encoding and decoding.

With reference to the foregoing description, a model training method of the present disclosure is described below. Referring to FIG. 4 , an example in which the present method is applied to a model training apparatus is used as an example for explanation. The model training device may be deployed on a server. An embodiment of a model training method in an embodiment of the present disclosure includes the following steps.

Step 201. The model training apparatus obtains a set of text to be trained and a set of image candidate regions to be trained, the set of text to be trained includes a first to-be-trained text and a second to-be-trained text, and the set of image candidate regions to be trained is a first a first image candidate region to be trained and a second image to-be-trained candidate region, wherein the first text to-be-trained and the first to-be-trained image candidate region have a matching relationship, and the first trained text and the second image candidate region to be trained does not have a matching relationship, the second to-be-trained text and the second to-be-trained image candidate region have a matching relationship, and the second to-be-trained text and the first to-be-trained image candidate region do not have a matching relationship.

In this embodiment, the model training apparatus first obtains a set of texts to be trained and a set of image candidate regions to be trained. The set of texts to be trained includes a first text to be trained and a second text to be trained. The set of image candidate regions to be trained includes a first image candidate region to be trained and a second image candidate region to be trained. In this case, the first text to be trained and the first to-be-trained image candidate region having a matching relationship are used as positive samples, and the second to-be-trained text to be trained and the second to-be-trained image candidate region having a matching relationship are used as positive samples. The first text-to-be-trained and second to-be-trained image candidate regions that do not have a matching relationship are used as speech samples, and the second text to-be-trained and second to-be-trained image candidate regions that do not have a matching relationship are used as speech samples.

It can be understood that the model training device is disposed on the server.

Step 202. The model training apparatus determines a target loss function according to the first to-be-trained text, the second to-be-trained text, the first to-be-trained image candidate region, and the second to-be-trained image candidate region.

In this embodiment, the model training apparatus learns a matching relationship between natural sentences and image candidate regions according to positive and negative samples to build a target loss function. The target loss function is mainly constructed to measure the similarity between the candidate image region and the natural sentence.

Step 203. The model training device trains an image region finding network model to be trained using a target loss function to obtain an image region finding network model, and the image region finding network model is an image candidate according to the text feature set and the enhanced semantic information. and determine a matching relationship between the region and the text to be found, wherein the enhanced semantic information and the image candidate region have a correspondence, and the text feature set and the text to be found have a correspondence.

In this embodiment, the model training apparatus uses the target loss function constructed to train the localization network model of the image region to be trained to further obtain the image region finding network model. The degree of matching between the image candidate region and the text to be found can be predicted using an image region finding network model. When the degree of matching is high, the degree of expression association is high.

In an embodiment of the present disclosure, a model training method is provided. The method includes first obtaining a set of text to be trained and a set of image candidate regions to be trained, wherein the set of text to be trained includes a first text to be trained and a second text to be trained, and the set of image candidate regions to be trained includes a first comprising an image candidate region to be trained and a second image candidate region to be trained, then a target according to the first text to be trained, the second text to be trained, the first image candidate region to be trained and the second image candidate region to be trained determining a loss function, and finally training an image region finding network model to be trained using the target loss function to obtain an image region finding network mode. In the above manner, the image region finding network model configured to determine the matching relationship between the image candidate region and the text may be obtained through training, and the target function used may determine the matching relationship between the text and the image candidate region through learning. It can be used to measure the similarity between the image candidate region and the text to obtain, thus improving the feasibility and operability of the solution.

Optionally, based on the above-described embodiment corresponding to FIG. 4 , in the first optional embodiment of the model training method provided in the embodiment of the present disclosure, by the model training apparatus, the first text to be trained, the second training Determining the target loss function according to the text to be trained, the first to-be-trained image candidate region, and the second to-be-trained image candidate region includes:

determining, by the model training device, a target loss function in the following manner,

는 타깃 손실 함수를 나타내고,

는 제1 훈련될 이미지 후보 영역을 나타내며,

는 제1 훈련될 텍스트를 나타내고,

는 제2 훈련될 이미지 후보 영역을 나타내며,

는 제2 훈련될 텍스트를 나타내고,

는 훈련될 데이터 쌍을 나타내며, max()는 최대값을 취함을 나타내고,

은 제1 파라미터 제어 가중치를 나타내며,

는 제2 파라미터 제어 가중치를 나타내고,

은 제1 미리 설정된 임계값을 나타내며,

는 제2 미리 설정된 임계값을 나타낸다.here,

represents the target loss function,

represents the first image candidate region to be trained,

represents the first to-be-trained text,

denotes the second to-be-trained image candidate region,

represents the second to-be-trained text,

denotes the data pair to be trained, max() denotes to take the maximum,

represents the first parameter control weight,

represents the second parameter control weight,

represents the first preset threshold,

denotes a second preset threshold value.

In this embodiment, a target loss function constructed using the model training apparatus is described, and the target loss function constructed based on the positive sample and the negative sample is expressed as:

는 양성 샘플을 나타낸다. 즉, 시맨틱 관계를 갖는 이미지 후보 영역과 자연어의 쌍을 나타낸다.

와

는 음성 샘플을 나타낸다. 즉, 이미지 후보 영역과 관련되지 않은 자연어의 쌍을 나타낸다.

는 양성 샘플이다. 대응하는 음성 샘플

는

에 대해 취해진다. 이러한 매칭 기능을 학습하면, 양성 샘플 사이의 매칭 관계가 음성 샘플 사이의 매칭 관계보다 더 높다.

는 양성 샘플이다. 대응하는 음성 샘플

은

에 대해 취해진다. 이러한 매칭 기능을 학습하면, 양성 샘플 사이의 매칭 관계가 음성 샘플 사이의 매칭 관계보다 더 높다.here,

indicates positive samples. That is, it represents a pair of an image candidate region and a natural language having a semantic relationship.

Wow

represents negative samples. That is, it represents a pair of natural language that is not related to the image candidate region.

is a positive sample. Corresponding voice samples

is

is taken about Learning this matching function, the matching relationship between positive samples is higher than that between negative samples.

is a positive sample. Corresponding voice samples

silver

is taken about Learning this matching function, the matching relationship between positive samples is higher than that between negative samples.

Next, in an embodiment of the present disclosure, a method of determining a target loss function according to a first to-be-trained text, a second to-be-trained text, a first to-be-trained image candidate region, and a second to-be-trained image candidate region is provided. In the above manner, the defined target loss function describes the matching relationship between the image and the natural language in two different directions. One direction is to associate image candidate regions with natural language, and the other direction is to associate natural language with image candidate regions. The main purpose of designing such a target loss function is to make the similarity between the semantically related image candidate region and the natural language pair higher than that of the semantically unrelated image candidate region and natural language pair, so that the accuracy of model training is higher. can improve

An apparatus for finding an image region in the present disclosure is described in detail below. Referring to FIG. 5 , FIG. 5 is a schematic diagram of an embodiment of an apparatus for finding an image region according to an embodiment of the present disclosure. The device 30 for finding an image area comprises:

an acquiring module 301, configured to acquire a set of image candidate regions in the image to be found, wherein the set of image candidate regions includes N image candidate regions, where N is an integer greater than or equal to one;

A generating module 302, configured to generate a set of region semantic information (ie, an image candidate region set of an image to be found) according to the set of image candidate regions obtained by the acquiring module 301 , the region semantic information set includes N region semantics information, wherein each region semantic information corresponds to one image candidate region (ie, each region semantic information of region semantic information corresponding to one image candidate region of an image candidate region set);

The acquiring module 301 is further configured to acquire an enhanced semantic information set corresponding to the regional semantic information set generated by the generating module 302 by using the GCN, wherein the enhanced semantic information set includes N enhanced semantic information wherein each enhanced semantic information corresponds to one region semantic information (that is, each enhanced semantic information in the enhanced semantic information set corresponding to one region semantic information in the region semantic information set), and the GCN corresponds to the various region semantics is configured to establish associations between information;

The obtaining module 301 is further configured to obtain a set of text features corresponding to the text to be found, wherein the text to be found includes T words, and the set of text features includes T word features, wherein each word includes: corresponds to one word feature, where T is greater than or equal to 1 (ie, each word in the text to be found corresponding to a one-word feature in the set of text features),

The obtaining module 301 is further configured to obtain a matching degree between the text feature set (ie, the text feature set corresponding to the text to be found) and each enhanced semantic information by using the image region finding network model, the image region the find network model is configured to determine a matching relationship between the image candidate region and the text to be found; and

a determining module 303, configured to determine a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each enhanced semantic information obtained by the acquiring module 301

includes

In this embodiment, the acquiring module 301 acquires a set of image candidate regions of the image to be found, the image candidate region set includes N image candidate regions, N is an integer greater than or equal to 1, and the generating module ( 302 generates a region semantic information set according to the image candidate region set obtained by the acquiring module 301, the region semantic information set includes N regions semantic information, each region semantic information is one image candidate corresponding to the region, the acquiring module 301 acquires the enhanced semantic information set corresponding to the region semantic information set generated by the generating module 302 by using the GCN, the enhanced semantic information includes N enhanced semantic information wherein each enhanced semantic information corresponds to one region semantic information, and is configured to establish an association relationship between various region semantic information, the obtaining module 301 acquires a set of text features corresponding to the text to be found, The text to be searched includes T words, the text feature set includes T word features, each word corresponds to one word feature, T is an integer greater than or equal to 1, and the acquiring module 301 . uses the image region finding network model to obtain a matching degree between the text feature set and each enhanced semantic information, the image region finding network model is configured to determine a matching relationship between the image candidate region and the text to be found, determining The module 303 determines a target image candidate region from the image candidate region according to a matching degree between the text feature set and each enhanced semantic information obtained by the acquiring module 301 .

In an embodiment of the present disclosure, an apparatus for finding an image region is provided. The apparatus first obtains a set of image candidate regions from the image to be found, the set of image candidate regions comprising N image candidate regions, and then generates a set of region semantic information according to the next set of image candidate regions, and ― each region semantic information corresponds to one image candidate region, and then obtains an enhanced semantic information set corresponding to the regional semantic information set using the GCN, each enhanced semantic information corresponds to one region semantic information, and the GCN is a multi-region configured to build an association relationship between the semantic information, further obtaining a set of text features corresponding to the text to be found, and then matching between the set of text features and each enhanced semantic information using the image region finding network model obtain a degree, and finally determine a target image candidate area from the image candidate area set according to the matching degree between the text feature set and each enhanced semantic information. In the above-described manner, the semantic representation between image candidate regions can be effectively improved by using GCN, and spatial relationships between image candidate regions are considered, so that the accuracy of finding image regions can be increased, thereby improving image understanding ability. can

Optionally, based on the above-described embodiment corresponding to FIG. 5 , in another embodiment of the apparatus 30 for finding an image region provided in this embodiment of the present disclosure,

The generating module 302 specifically uses CNN to obtain region semantic information corresponding to each image candidate region, wherein the image candidate region includes region information, and the region information includes location information of an image candidate region in the image to be found. and size information of the image candidate area ―,

and when region semantic information corresponding to the N image candidate regions is obtained, generate a region semantic information set according to the N region semantic information.

Next, in an embodiment of the present disclosure, a method of generating a region semantic information set is provided. Region semantic information corresponding to the image candidate region is obtained using CNN. The image candidate region includes region information. The region information includes position information of the image candidate region and size information of the image candidate region in the image to be found. The region semantic information set is generated according to the N region semantic information when region semantic information corresponding to the N image candidate regions is obtained. In the above-described manner, region semantic information of each image candidate region may be extracted using CNN. CNN is a feed-forward neural network. The artificial neurons of CNNs can respond to neighboring units in the partial coverage area, so they have excellent performance for large-scale image processing. In this way, the accuracy of information extraction is improved.

Optionally, based on the above-described embodiment corresponding to FIG. 5 , in another embodiment of the apparatus 30 for finding an image region provided in this embodiment of the present disclosure,

The obtaining module 301 is specifically configured to obtain the first region semantic information and the second region semantic information from the region semantic information set, wherein the first region semantic information is any one region semantic information in the region semantic information set, and the second region semantic information is The region semantic information is any one region semantic information in the region semantic information set ―,

obtaining the strength of a connection edge between the first region semantic information and the second region semantic information;

Normalize the strength of the connecting edge between the first region semantic information and the second region semantic information to obtain a normalized strength,

determine a target connection matrix according to the normalized strength between the various domain semantic information in the domain semantic information set;

and determine an enhanced semantic information set corresponding to the target connectivity matrix using the GCN.

Next, in an embodiment of the present disclosure, a method of obtaining an enhanced semantic information set using GCN is provided. First, the first region semantic information and the second region semantic information are obtained from the region semantic information set. Then, the strength of the connecting edge between the first region semantic information and the second region semantic information is obtained. Next, the strength of the connecting edge between the first region semantic information and the second region semantic information is normalized to obtain a normalized strength. Then, a target connection matrix is determined according to the normalized strength between the various region semantic information in the region semantic information set. Finally, the set of enhanced semantic information corresponding to the target connectivity matrix is determined using the GCN. In the above-described manner, a semantic relationship between image candidate regions is established using GCN. In this way, the spatial information and the semantic relationship are sufficiently taken into account, thereby improving the image-based finding performance.

Optionally, based on the above-described embodiment corresponding to FIG. 5 , in another embodiment of the apparatus 30 for finding an image region provided in this embodiment of the present disclosure,

The acquiring module 301 specifically generates a connection matrix according to the normalized strength between the various region semantic information in the region semantic information set,

and generate a target connection matrix according to the connection matrix and the unit matrix.

Next, in an embodiment of the present disclosure, a method of determining a target connection matrix according to normalized strengths between various region semantic information in a region semantic information set is provided. That is, the connection matrix is first generated according to the normalized strength between the various region semantic information of the region semantic information set. Then, a target connection matrix is generated according to the connection matrix and the unit matrix. In the above-described manner, through the process measurement of normalization, the absolute value of a value in the physical system can be converted into a relative value relationship, so that the calculation can be simplified and the size can be reduced. Further, to further enhance the information, the unit matrix is added to the corresponding connection matrix to form a target connection matrix.

Optionally, based on the above-described embodiment corresponding to FIG. 5 , in another embodiment of the apparatus 30 for finding an image region provided in this embodiment of the present disclosure,

The acquiring module is specifically configured to calculate the set of enhanced semantic information in the following way,

는 GCN의 k 번째 계층에 대응하는 i 번째 향상된 시맨틱 정보를 나타내고,

는 GCN의 k-1 번째 계층에 대응하는 j 번째 향상된 시맨틱 정보를 나타내며,

는 GCN의 k 번째 계층의 제1 네트워크 파라미터를 나타내고,

는 GCN의 k 번째 계층의 제2 네트워크 파라미터를 나타내며,

는 j 번째 노드가 i 번째 노드의 이웃 노드임을 나타내고,

는 타깃 연결 매트릭스의 요소를 나타낸다. here,

represents the i-th enhanced semantic information corresponding to the k-th layer of GCN,

represents the j-th enhanced semantic information corresponding to the k-1 th layer of GCN,

represents the first network parameter of the k-th layer of GCN,

represents the second network parameter of the k-th layer of GCN,

indicates that the j-th node is a neighbor of the i-th node,

represents the elements of the target connection matrix.

In addition, in an embodiment of the present disclosure, a specific manner of an enhanced semantic information set corresponding to a target connection matrix determined using GCN is provided. In the above-described manner, since a specific calculation method is provided for the GCN-based calculation, it is possible to improve the feasibility and operability of the solution.

Optionally, based on the above-described embodiment corresponding to FIG. 5 , in another embodiment of the apparatus 30 for finding an image region provided in this embodiment of the present disclosure,

The acquiring module 301 specifically acquires the text to be found,

obtain a text vector sequence according to the text to be found, the text vector sequence comprising T word vectors, each word vector corresponding to a word;

encode each word vector in the text vector sequence to obtain a text feature,

and when text features corresponding to the T word vectors are obtained, generate a set of text features according to the T text features.

Next, in an embodiment of the present disclosure, a method for obtaining a text feature set is provided. That is, the text to be found is obtained first. Then, a text vector sequence is obtained according to the text to be searched for. The text vector sequence contains T word vectors. Each word vector corresponds to one word. Next, each word vector of the text vector sequence is encoded to obtain text features. A text feature set is generated according to the T text features when text features corresponding to the T word vectors are obtained. In the above-described manner, the text to be found can be expressed in the form of features to further facilitate subsequent model prediction, thereby improving the feasibility and operability of the solution.

Optionally, based on the above-described embodiment corresponding to FIG. 5 , in another embodiment of the apparatus 30 for finding an image region provided in this embodiment of the present disclosure,

The acquiring module 301 is specifically configured to acquire the text feature in the following way,

는 텍스트 특징 세트에서 t 번째 텍스트 특징을 나타내고,

는 LSTM 네트워크를 사용하여 인코딩을 수행하는 것을 나타내며,

는 텍스트 벡터 시퀀스에서 t 번째 단어 벡터를 나타내고,

는 텍스트 특징 세트에서 (t-1) 번째 텍스트 특징을 나타낸다.here,

denotes the t-th text feature in the text feature set,

indicates that encoding is performed using the LSTM network,

denotes the t-th word vector in the text vector sequence,

denotes the (t-1)-th text feature in the text feature set.

Next, in an embodiment of the present disclosure, a method of obtaining a text feature is provided. That is, the RNN of the LSTM structure is used to encode the word vector. In the above-described manner, if the network of the LSTM structure is used, the problem of gradient loss due to the gradual reduction of the backpropagation process can be solved. Specifically, in language processing tasks, LSTMs are suitable for handling temporal height related problems, such as machine translation, dialog generation, and encoding and decoding.

The model training apparatus of the present disclosure is described in detail below. Referring to FIG. 6 , FIG. 6 is a schematic diagram of an embodiment of a model training apparatus according to an embodiment of the present disclosure. Model training device 40,

an acquiring module 401 configured to acquire a set of text to be trained and a set of image candidate regions to be trained, wherein the set of text to be trained includes a first text to be trained and a second text to be trained, and the set of image candidate regions to be trained includes the first an image candidate region to be trained and a second image candidate region to be trained, wherein the first text to be trained and the first image to be trained region have a matching relationship, and the first trained text and the second image candidate region to be trained does not have a matching relationship, the second to-be-trained text and the second to-be-trained image candidate region have a matching relationship, and the second to-be-trained text and the first to-be-trained image candidate region do not have a matching relationship;

A determining module 402, configured to determine a target loss function according to the first text to be trained, the second text to be trained, the first image to be trained region, and the second image to be trained region obtained by the acquiring module 401 . ; and

a training model 403 configured to train an image region finding network model to be trained using the target loss function determined by the determining module 402 to obtain an image region finding network model, wherein the image region finding network model includes text features determine a matching relationship between the image candidate region and the text to be found according to the set and the enhanced semantic information, wherein the enhanced semantic information and the image candidate region have a correspondence, and the text feature set and the text to be found have a correspondence,

includes

In this embodiment, the acquiring module 401 acquires a set of text to be trained and a set of image candidate regions to be trained, wherein the set of text to be trained includes a first text to be trained and a second text to be trained, and an image candidate to be trained The set of regions includes a first candidate image to be trained region and a second candidate image to be trained region, wherein the first text to be trained and the first candidate image to be trained region have a matching relationship, and the first text to be trained and the second The image candidate region to be trained does not have a matching relationship, the second to-be-trained text and the second to-be-trained image candidate region have a matching relationship, and the second to-be-trained text and the first to-be-trained image candidate region have a matching relationship. do not have -, the determining module 402 determines a target according to the first text-to-be-trained, second to-be-trained text, first to-be-trained image candidate region and second to-be-trained image candidate region obtained by the acquiring module 401 . determine a loss function, the training module 403 trains the image region finding network model to be trained using the target loss function determined by the determining module 402 to obtain an image region finding network model, The network model is configured to determine a matching relationship between the image candidate region and the text to be found according to the text feature set and the enhanced semantic information, the enhanced semantic information and the image candidate region have a correspondence, the text feature set and the text to be found are have a corresponding relationship.

In an embodiment of the present disclosure, a model training apparatus is provided. The model training apparatus first obtains a set of text to be trained and a set of image candidate regions to be trained, wherein the set of text to be trained includes a first text to be trained and a second text to be trained, and the set of image candidate regions to be trained includes a first training comprising an image candidate region to be trained and a second image candidate region to be trained, then target loss according to the first text to be trained, the second text to be trained, the first image candidate region to be trained and the second image candidate region to be trained A function is determined, and finally, an image region finding network model to be trained is trained using the target loss function to obtain an image region finding network model. In the above manner, the image region finding network model configured to determine the matching relationship between the image candidate region and the text may be obtained through training, and the target function used may determine the matching relationship between the text and the image candidate region through learning. It can be used to measure the similarity between the image candidate region and the text to obtain, thus improving the feasibility and operability of the solution.

Optionally, based on the above-described embodiment corresponding to FIG. 6 , in another embodiment of the model training apparatus 40 provided in this embodiment of the present disclosure,

The determining module 402 is specifically configured to determine the target loss function in the following manner,

는 타깃 손실 함수를 나타내고,

는 제1 훈련될 이미지 후보 영역을 나타내며,

는 제1 훈련될 텍스트를 나타내고,

는 제2 훈련될 이미지 후보 영역을 나타내며,

는 제2 훈련될 텍스트를 나타내고,

는 훈련될 데이터 쌍을 나타내며, max()는 최대값을 취함을 나타내고,

은 제1 파라미터 제어 가중치를 나타내며,

는 제2 파라미터 제어 가중치를 나타내고,

은 제1 미리 설정된 임계값을 나타내며,

는 제2 미리 설정된 임계값을 나타낸다.here,

represents the target loss function,

represents the first image candidate region to be trained,

represents the first to-be-trained text,

denotes the second to-be-trained image candidate region,

represents the second to-be-trained text,

denotes the data pair to be trained, max() denotes to take the maximum,

represents the first parameter control weight,

represents the second parameter control weight,

represents the first preset threshold,

denotes a second preset threshold value.

Next, in an embodiment of the present disclosure, a method of determining a target loss function according to a first to-be-trained text, a second to-be-trained text, a first to-be-trained image candidate region, and a second to-be-trained image candidate region is provided. In the above manner, the defined target loss function describes the matching relationship between the image and the natural language in two different directions. One direction is to associate image candidate regions with natural language, and the other direction is to associate natural language with image candidate regions. The main purpose of designing such a target loss function is to make the similarity between the semantically related image candidate region and the natural language pair higher than that of the semantically unrelated image candidate region and natural language pair, so that the accuracy of model training is higher. can improve

The embodiment of the present disclosure further provides another apparatus for finding an image region, as shown in FIG. 7 , and only parts related to the embodiment of the present disclosure are shown for convenience of description. For specific technical matters not disclosed, refer to the method part of the embodiment of the present disclosure. The device may be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), and an on-board computer, and a terminal device that is a mobile phone is used as an example.

7 is a block diagram of a partial structure of a mobile phone related to a terminal device according to an embodiment of the present disclosure. Referring to FIG. 7 , the mobile phone includes a radio frequency (RF) circuit 510 , a memory 520 , an input unit 530 , a display unit 540 , a sensor 550 , an audio circuit 560 , and Wi-Fi. It includes components such as a (Wireless Fidelity, Wi-Fi) module, a processor 580 and a power supply 590 . For those skilled in the art, the structure of the mobile phone shown in Fig. 7 does not constitute a limitation for the mobile phone, and the mobile phone may include more or fewer components than those shown in the drawing, or some components may be combined. or other component arrangements may be used.

Hereinafter, components of the mobile phone will be described in detail with reference to FIG. 7 .

The RF circuitry 510 may be configured to receive and transmit signals during an information reception and transmission process or a call process. Specifically, the RF circuit receives the downlink information from the base station, then passes the downlink information to the processor 580 for processing, and transmits the designed uplink data to the base station. In general, the RF circuitry 510 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), and a duplexer. In addition, the RF circuit 510 may communicate with a network and other devices through wireless communication. Wireless communication includes Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), Email, Short Messaging (SMS) Service), and the like, may use any communication standard or protocol, including but not limited to. .

Memory 520 may be configured to store software programs and modules. The processor 580 executes software programs and modules stored in the memory 520 to implement various functional applications and data processing of the mobile phone. The memory 520 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, an application program required for at least one function (a sound reproduction function, an image display function, etc.). The data storage area may store data (audio data, address book, etc.) generated according to the use of the mobile phone. Memory 520 may also include high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory, or other volatile solid-state storage device.

The input unit 530 may be configured to receive input number or text information, and to generate a keyboard signal input related to user settings and function control of the mobile phone. Specifically, the input unit 530 may include a touch panel 531 and another input device 532 . Touch panel 531 , which may also be referred to as a touch screen, is configured for a user's touch manipulation on or near the touch panel (eg, on or near touch panel 531 using any suitable object or accessory such as a finger or stylus). A user's operation near the touch panel 531) may be collected, and a corresponding connected device may be driven according to a preset program. Optionally, the touch panel 531 may include two parts, namely, a touch detection device and a touch controller. The touch detection device detects a user's touch position, detects a signal generated by a touch manipulation, and transmits the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts the touch information into touch point coordinates, and transmits the touch point coordinates to the processor 580 . Also, the touch controller may receive and execute a command transmitted from the processor 580 . In addition, the touch panel 531 may be implemented using various types, such as a resistive type, a capacitive type, an infrared type, and a surface acoustic wave type. In addition to the touch panel 531 , the input unit 530 may further include another input device 532 . Specifically, other input devices 532 may include, but are not limited to, one or more of a physical keyboard, function keys (eg, volume control keys or switch keys), a trackball, a mouse, and a joystick.

The display unit 540 may be configured to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 540 may include a display panel 541 . Optionally, the display panel 541 may be configured using a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Also, the touch panel 531 may cover the display panel 541 . After detecting a touch manipulation on or near the touch panel 531 , the touch panel passes the touch manipulation to the processor 580 to determine the type of touch event. Then, the processor 580 provides a visual output corresponding to the display panel 541 according to the type of the touch event. Although the touch panel 531 and the display panel 541 in FIG. 7 are used as two separate parts to implement the input/output function of the mobile phone, in some embodiments, the touch panel 531 and the display panel 541 are It can be integrated to implement the input/output function of the mobile phone.

The mobile phone may further include at least one sensor 550 such as an optical sensor, a motion sensor, and other sensors. Specifically, the optical sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor may adjust the brightness of the display panel 541 according to the brightness of the ambient light. The proximity sensor may turn off the display panel 541 and/or the backlight when the phone is moved to the ear. As a type of motion sensor, the accelerometer can detect the magnitude of acceleration in various directions (usually three axes), and can detect the magnitude and direction of gravity when stationary, and the posture of the mobile phone (for example, , switching between landscape and portrait orientations, related games and magnetometer posture correction), vibration recognition-related functions (eg, pedometer and knock), and the like. Other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors that may be configured in a mobile phone are not further described herein.

Audio circuitry 560 , loudspeaker 561 and microphone 562 may provide an audio interface between the user and the mobile phone. The audio circuit 560 may convert the received audio data into an electrical signal and transmit the electrical signal to the speaker 561 . The speaker 561 converts the electrical signal into a sound signal for output. Meanwhile, the microphone 562 converts the collected sound signal into an electrical signal. The audio circuit 560 receives the electrical signal, converts the electrical signal into audio data, and outputs the audio data to the processor 580 for processing. The processor then uses the RF circuit 510 to transmit the audio data to, for example, another mobile phone, or outputs the audio data to the memory 520 for further processing.

Wi-Fi is a short-range wireless transmission technology. The mobile phone may use the Wi-Fi module 570 to help a user receive and send email, navigate web pages, access stream media, and the like. It provides users with wireless broadband Internet access. Although FIG. 7 shows the Wi-Fi module 570, it is understood that the Wi-Fi module is not an essential component of the mobile phone, and the Wi-Fi module is not required unless the scope of the essence of the present disclosure is changed. It can be seen that it can be omitted accordingly.

The processor 580 is the control center of the mobile phone and connects to the various parts of the overall mobile phone using various interfaces and lines. By operating or executing a software program and/or module stored in the memory 520 and calling data stored in the memory 520, the processor executes various functions of the mobile phone and performs data processing, thereby monitoring the entire mobile phone. can Optionally, processor 580 may include one or more processing units. Optionally, processor 580 may incorporate an application processor and a modem processor. The application processor mainly processes the operating system, user interface, application programs, and the like. The modem processor primarily handles wireless communications. It will be appreciated that the modem described above may not be integrated into the processor 580 .

The mobile phone further includes a power supply 590 (eg, a battery) for powering the components. Optionally, the power supply can be logically coupled to the processor 580 using a power management system, so that the power management system can be used to implement functions such as charging, discharging, and power consumption management.

Although not shown in the drawings, the mobile phone may further include a camera, a Bluetooth module, and the like, which are not further described herein.

In this embodiment of the present disclosure, the processor 580 included in the terminal device,

a function to obtain a set of image candidate regions in the image to be found, wherein the set of image candidate regions includes N image candidate regions, where N is an integer greater than or equal to one;

A function of generating a set of region semantic information (i.e., a set of image candidate regions of the image to be found) according to a set of image candidate regions, the region semantic information set comprising N region semantic information, each region semantic information being one image corresponding to a candidate region (i.e., each region semantic information of region semantic information corresponding to one image candidate region in a set of image candidate regions);

GCN is used to obtain an enhanced semantic information set corresponding to the region semantic information set, the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information is one region semantic information (that is, region semantic information each enhanced semantic information of the set of enhanced semantic information corresponding to one region semantic information of the set, the GCN is configured to establish an association relationship between the various region semantic information;

obtain a set of text features corresponding to the text to be found, the text to be found comprises T words, the text feature set comprises T word features, each word corresponding to one word feature, and T is greater than or equal to 1 (ie, each word in the text to be found corresponding to a one-word feature in the set of text features),

The image region finding network model is used to obtain the degree of matching between the text feature set (that is, the text feature set corresponding to the text to be found) and each enhanced semantic information, and the image region finding network model is the image candidate region and the to be found. configured to determine a matching relationship between texts; and

The ability to determine a target image candidate region from a set of image candidate regions according to the degree of matching between the text feature set and each enhanced semantic information.

has additionally

Optionally, the processor 580 specifically:

Acquiring region semantic information corresponding to each image candidate region using CNN—The image candidate region includes region information, and the region information includes location information of an image candidate region and size information of an image candidate region in an image to be found. including ―; and

When region semantic information corresponding to the N image candidate regions is obtained, generating a region semantic information set according to the N region semantic information;

is configured to perform

Optionally, the processor 580 specifically:

obtaining first region semantic information and second region semantic information from the region semantic information set, wherein the first region semantic information is any one region semantic information in the region semantic information set, and the second region semantic information is region semantic information any one area semantic information in the set;

obtaining a strength of a connection edge between the first region semantic information and the second region semantic information;

normalizing the strength of a connecting edge between the first region semantic information and the second region semantic information to obtain a normalized strength;

determining a target connection matrix according to normalized strengths between various region semantic information of the region semantic information set; and

Determining an enhanced semantic information set corresponding to a target connectivity matrix using GCN

is configured to perform

Optionally, the processor 580 specifically:

generating a connection matrix according to normalized strengths between various region semantic information in the region semantic information set; and

generating a target connection matrix according to the connection matrix and the unit matrix;

is configured to perform

Optionally, the processor 580 specifically:

is configured to perform the steps of calculating an enhanced set of semantic information in the following manner:

는 GCN의 k 번째 계층에 대응하는 i 번째 향상된 시맨틱 정보를 나타내고,

는 GCN의 k-1 번째 계층에 대응하는 j 번째 향상된 시맨틱 정보를 나타내며,

는 GCN의 k 번째 계층의 제1 네트워크 파라미터를 나타내고,

는 GCN의 k 번째 계층의 제2 네트워크 파라미터를 나타내며,

는 j 번째 노드가 i 번째 노드의 이웃 노드임을 나타내고,

는 타깃 연결 매트릭스의 요소를 나타낸다. here,

represents the i-th enhanced semantic information corresponding to the k-th layer of GCN,

represents the j-th enhanced semantic information corresponding to the k-1 th layer of GCN,

represents the first network parameter of the k-th layer of GCN,

represents the second network parameter of the k-th layer of GCN,

indicates that the j-th node is a neighbor of the i-th node,

represents the elements of the target connection matrix.

Optionally, the processor 580 specifically:

obtaining the text to be found;

obtaining a text vector sequence according to the text to be found, the text vector sequence comprising T word vectors, each word vector corresponding to one word;

encoding each word vector in the text vector sequence to obtain a text feature; and

when text features corresponding to the T word vectors are obtained, generating a set of text features according to the T text features;

is configured to perform

Optionally, the processor 580 specifically:

is configured to perform the steps of obtaining text features in the following manner,

는 텍스트 특징 세트에서 t 번째 텍스트 특징을 나타내고,

는 LSTM 네트워크를 사용하여 인코딩을 수행하는 것을 나타내며,

는 텍스트 벡터 시퀀스에서 t 번째 단어 벡터를 나타내고,

는 텍스트 특징 세트에서 (t-1) 번째 텍스트 특징을 나타낸다. here,

denotes the t-th text feature in the text feature set,

indicates that encoding is performed using the LSTM network,

denotes the t-th word vector in the text vector sequence,

denotes the (t-1)-th text feature in the text feature set.

8 is a schematic structural diagram of a server according to an embodiment of the present disclosure. Server 600 can vary widely due to different configurations or performance, and includes one or more central processing units (CPUs) 622 (eg, one or more processors) and memory 632 and application programs 642 . ) and one or more storage media 630 (eg, one or more mass storage devices) for storing data 644 . Memory 632 and storage medium 630 may be temporary or permanent storage devices. The program stored in the storage medium 630 may include one or more modules (not shown), and each module may include a series of command operations for the server. In addition, the CPU 622 may be set to communicate with the storage medium 630 , and may perform a series of instruction operations of the storage medium 630 in the server 600 .

Server 600 may include one or more power supplies 626 , one or more wired or wireless network interfaces 650 , one or more input/output interfaces 658 , and/or Windows Server ^TM , Mac OS X ^TM , Unix ^TM , Linux . It may further include one or more operating systems 641 such as ^TM or FreeBSD ^TM.

The steps performed by the server in the above-described embodiment may be based on the server structure shown in FIG. 8 .

In this embodiment of the present invention, the CPU 622 included in the server,

a function to obtain a set of image candidate regions in the image to be found, wherein the set of image candidate regions includes N image candidate regions, where N is an integer greater than or equal to one;

A function of generating a set of region semantic information (i.e., a set of image candidate regions of the image to be found) according to a set of image candidate regions, the region semantic information set comprising N region semantic information, each region semantic information being one image corresponding to a candidate region (i.e., each region semantic information of region semantic information corresponding to one image candidate region in a set of image candidate regions);

GCN is used to obtain an enhanced semantic information set corresponding to the region semantic information set, the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information is one region semantic information (that is, region semantic information each enhanced semantic information of the set of enhanced semantic information corresponding to one region semantic information of the set, the GCN is configured to establish an association relationship between the various region semantic information;

obtain a set of text features corresponding to the text to be found, the text to be found comprises T words, the text feature set comprises T word features, each word corresponding to one word feature, and T is greater than or equal to 1 (ie, each word in the text to be found corresponding to a one-word feature in the set of text features),

The image region finding network model is used to obtain the degree of matching between the text feature set (that is, the text feature set corresponding to the text to be found) and each enhanced semantic information, and the image region finding network model is the image candidate region and the to be found. configured to determine a matching relationship between texts; and

The ability to determine a target image candidate region from a set of image candidate regions according to the degree of matching between the text feature set and each enhanced semantic information.

has additionally

Optionally, the CPU 622 specifically:

Acquiring region semantic information corresponding to each image candidate region using CNN—The image candidate region includes region information, and the region information includes location information of an image candidate region and size information of an image candidate region in an image to be found. including ―; and

When region semantic information corresponding to the N image candidate regions is obtained, generating a region semantic information set according to the N region semantic information;

is configured to perform

Optionally, the CPU 622 specifically:

obtaining first region semantic information and second region semantic information from the region semantic information set, wherein the first region semantic information is any one region semantic information in the region semantic information set, and the second region semantic information is region semantic information any one area semantic information in the set;

obtaining a strength of a connection edge between the first region semantic information and the second region semantic information;

normalizing the strength of a connecting edge between the first region semantic information and the second region semantic information to obtain a normalized strength;

determining a target connection matrix according to normalized strengths between various region semantic information of the region semantic information set; and

Determining an enhanced semantic information set corresponding to a target connectivity matrix using GCN

is configured to perform

Optionally, the CPU 622 specifically:

generating a connection matrix according to normalized strengths between various region semantic information in the region semantic information set; and

generating a target connection matrix according to the connection matrix and the unit matrix;

is configured to perform

Optionally, the CPU 622 specifically:

is configured to perform the steps of calculating an enhanced set of semantic information in the following manner:

는 GCN의 k 번째 계층에 대응하는 i 번째 향상된 시맨틱 정보를 나타내고,

는 GCN의 k-1 번째 계층에 대응하는 j 번째 향상된 시맨틱 정보를 나타내며,

는 GCN의 k 번째 계층의 제1 네트워크 파라미터를 나타내고,

는 GCN의 k 번째 계층의 제2 네트워크 파라미터를 나타내며,

는 j 번째 노드가 i 번째 노드의 이웃 노드임을 나타내고,

는 타깃 연결 매트릭스의 요소를 나타낸다. here,

represents the i-th enhanced semantic information corresponding to the k-th layer of GCN,

represents the j-th enhanced semantic information corresponding to the k-1 th layer of GCN,

represents the first network parameter of the k-th layer of GCN,

represents the second network parameter of the k-th layer of GCN,

indicates that the j-th node is a neighbor of the i-th node,

represents the elements of the target connection matrix.

Optionally, the CPU 622 specifically:

obtaining the text to be found;

obtaining a text vector sequence according to the text to be found, the text vector sequence comprising T word vectors, each word vector corresponding to one word;

encoding each word vector in the text vector sequence to obtain a text feature; and

when text features corresponding to the T word vectors are obtained, generating a set of text features according to the T text features;

is configured to perform

Optionally, the CPU 622 specifically:

is configured to perform the steps of obtaining text features in the following manner,

는 텍스트 특징 세트에서 t 번째 텍스트 특징을 나타내고,

는 LSTM 네트워크를 사용하여 인코딩을 수행하는 것을 나타내며,

는 텍스트 벡터 시퀀스에서 t 번째 단어 벡터를 나타내고,

는 텍스트 특징 세트에서 (t-1) 번째 텍스트 특징을 나타낸다. here,

denotes the t-th text feature in the text feature set,

indicates that encoding is performed using the LSTM network,

denotes the t-th word vector in the text vector sequence,

denotes the (t-1)-th text feature in the text feature set.

In this embodiment of the present disclosure, the CPU 622 included in the server,

a function of obtaining a set of text to be trained and a set of image candidate regions to be trained, wherein the set of text to be trained includes a first text to be trained and a second text to be trained, and the set of image candidate regions to be trained is a first image candidate region to be trained. and a second image candidate region to be trained, wherein the first text to be trained and the first image candidate region to be trained have a matching relationship, and the first trained text and the second image candidate region to be trained have a matching relationship not, the second text to be trained and the second to-be-trained image candidate region have a matching relationship, and the second to-be-trained text and the first to-be-trained image candidate region do not have a matching relationship;

determining a target loss function according to the first text to be trained, the second text to be trained, the first to-be-trained image candidate region, and the second to-be-trained image candidate region; and

The ability to train an image region finding network model to be trained using a target loss function to obtain an image region finding network model — the image region finding network model is constructed between image candidate regions and the text to be found according to a set of text features and enhanced semantic information. configured to determine a matching relationship of , wherein the enhanced semantic information and the image candidate region have a correspondence, and the text feature set and the text to be found have a correspondence —

have additional

Optionally, the CPU 622 specifically:

is configured to perform the steps of determining a target loss function in the following manner:

는 타깃 손실 함수를 나타내고,

는 제1 훈련될 이미지 후보 영역을 나타내며,

는 제1 훈련될 텍스트를 나타내고,

는 제2 훈련될 이미지 후보 영역을 나타내며,

는 제2 훈련될 텍스트를 나타내고,

는 훈련될 데이터 쌍을 나타내며, max()는 최대값을 취함을 나타내고,

은 제1 파라미터 제어 가중치를 나타내며,

는 제2 파라미터 제어 가중치를 나타내고,

은 제1 미리 설정된 임계값을 나타내며,

는 제2 미리 설정된 임계값을 나타낸다.here,

represents the target loss function,

represents the first image candidate region to be trained,

represents the first to-be-trained text,

denotes the second to-be-trained image candidate region,

represents the second to-be-trained text,

denotes the data pair to be trained, max() denotes to take the maximum,

represents the first parameter control weight,

represents the second parameter control weight,

represents the first preset threshold,

denotes a second preset threshold value.

It is clear to those skilled in the art that for convenience and conciseness of description, corresponding processes in the above-described method embodiments may be referred to for specific working processes of the above-described systems, apparatuses and units, and details are not described herein again. can be understood

In the embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other ways. For example, the described device embodiments are exemplary only. For example, unit division is only logical function division, and may be another division in actual implementation. For example, a plurality of units or components may be combined or integrated into other systems, or some features may be ignored or not performed. Also, the mutual couplings or direct couplings or communication connections displayed or discussed may be implemented using some interfaces. An indirect coupling or communication connection between devices or units may be implemented as an electrical, mechanical, or other type of connection.

A unit described as a separate component may or may not be physically separated, and a component displayed as a unit may or may not be a physical unit, and may be located in one location or distributed over a plurality of network units . Some or all of these units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware, or may be implemented in the form of a software functional unit.

When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure may be implemented in the form of a software product essentially, or a part contributing to the prior art, or all or a part of the technical solution. A computer software product is stored in a storage medium and contains several instructions for instructing a computer device (which may be a PC, a server, or a network device) to perform all or part of the method steps described in the embodiments of the present disclosure. include The aforementioned storage medium includes any medium capable of storing a program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a RAM, a magnetic disk, or an optical disk.

Embodiments of the present disclosure further provide a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, cause a computer to use a method for locating an image region provided in the above-described embodiments. Lets do any possible implementation.

Optionally, the instructions stored on the computer-readable storage medium include:

generating region semantic information according to the image candidate region set of the image to be found, each region semantic information of the region semantic information set corresponding to one image candidate region of the image candidate region set;

obtaining an enhanced semantic information set corresponding to the region semantic information set using the GCN, wherein each enhanced semantic information of the enhanced semantic information set corresponds to one region semantic information of the region semantic information set, and the GCN is configured to establish associations between information;

obtaining a degree of matching between each enhanced semantic information and a set of text features corresponding to the text to be found using the image region finding network model—the image region finding network model determines the matching relationship between the image candidate region and the text to be found configured to determine, wherein each word of the text to be found corresponds to a word of the set of text features; and

determining a target image candidate region from the image candidate region set according to a degree of matching between the text feature set and each enhanced semantic information;

is configured to perform

Optionally, the instructions stored on the computer-readable storage medium include:

Acquiring region semantic information corresponding to each image candidate region using CNN—The image candidate region includes region information, and the region information includes location information of an image candidate region and size information of an image candidate region in an image to be found. including ―; and

When region semantic information corresponding to the N image candidate regions is obtained, generating a region semantic information set according to the N region semantic information;

is configured to perform

Optionally, the instructions stored on the computer-readable storage medium include:

obtaining first region semantic information and second region semantic information from the region semantic information set, wherein the first region semantic information is any one region semantic information in the region semantic information set, and the second region semantic information is region semantic information any one area semantic information in the set;

obtaining a strength of a connection edge between the first region semantic information and the second region semantic information;

normalizing the strength of a connecting edge between the first region semantic information and the second region semantic information to obtain a normalized strength;

determining a target connection matrix according to normalized strengths between various region semantic information of the region semantic information set; and

Determining an enhanced semantic information set corresponding to a target connectivity matrix using GCN

is configured to perform

Optionally, the instructions stored on the computer-readable storage medium include:

generating a connection matrix according to normalized strengths between various region semantic information in the region semantic information set; and

generating a target connection matrix according to the connection matrix and the unit matrix;

is configured to perform

Optionally, the instructions stored on the computer-readable storage medium include:

is configured to perform the steps of calculating an enhanced set of semantic information in the following manner:

는 GCN의 k 번째 계층에 대응하는 i 번째 향상된 시맨틱 정보를 나타내고,

는 GCN의 k-1 번째 계층에 대응하는 j 번째 향상된 시맨틱 정보를 나타내며,

는 GCN의 k 번째 계층의 제1 네트워크 파라미터를 나타내고,

는 GCN의 k 번째 계층의 제2 네트워크 파라미터를 나타내며,

는 j 번째 노드가 i 번째 노드의 이웃 노드임을 나타내고,

는 타깃 연결 매트릭스의 요소를 나타낸다. here,

represents the i-th enhanced semantic information corresponding to the k-th layer of GCN,

represents the j-th enhanced semantic information corresponding to the k-1 th layer of GCN,

represents the first network parameter of the k-th layer of GCN,

represents the second network parameter of the k-th layer of GCN,

indicates that the j-th node is a neighbor of the i-th node,

represents the elements of the target connection matrix.

Optionally, the instructions stored on the computer-readable storage medium include:

obtaining the text to be found;

obtaining a text vector sequence according to the text to be found, the text vector sequence comprising T word vectors, each word vector corresponding to one word, and T being greater than or equal to one;

encoding each word vector in the text vector sequence to obtain a text feature; and

when text features corresponding to the T word vectors are obtained, generating a set of text features according to the T text features;

is configured to perform

Optionally, the instructions stored on the computer-readable storage medium include:

is configured to perform the steps of obtaining text features in the following manner,

는 텍스트 특징 세트에서 t 번째 텍스트 특징을 나타내고,

는 LSTM 네트워크를 사용하여 인코딩을 수행하는 것을 나타내며,

는 텍스트 벡터 시퀀스에서 t 번째 단어 벡터를 나타내고,

는 텍스트 특징 세트에서 (t-1) 번째 텍스트 특징을 나타낸다. here,

denotes the t-th text feature in the text feature set,

indicates that encoding is performed using the LSTM network,

denotes the t-th word vector in the text vector sequence,

denotes the (t-1)-th text feature in the text feature set.

Embodiments of the present disclosure further provide a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, cause the computer to perform any possible of the model training methods provided in the above-described embodiments. Let the implementation

Optionally, the instructions stored on the computer-readable storage medium include:

obtaining a set of text to be trained and a set of image candidate regions to be trained, wherein the set of text to be trained includes a first text to be trained and a second text to be trained, and the set of image candidate regions to be trained includes a first image candidate region to be trained. and a second image candidate region to be trained, wherein the first text to be trained and the first image candidate region to be trained have a matching relationship, and the first trained text and the second image candidate region to be trained have a matching relationship not, the second text to be trained and the second to-be-trained image candidate region have a matching relationship, and the second to-be-trained text and the first to-be-trained image candidate region do not have a matching relationship;

determining a target loss function according to the first to-be-trained text, the second to-be-trained text, the first to-be-trained image candidate region, and the second to-be-trained image candidate region; and

training an image region finding network model to be trained using the target loss function to obtain an image region finding network model — the image region finding network model is formed between the image candidate region and the text to be found according to a set of text features and enhanced semantic information. configured to determine a matching relationship of , wherein the enhanced semantic information and the image candidate region have a correspondence, and the text feature set and the text to be found have a correspondence —

is configured to perform

Optionally, the instructions stored on the computer-readable storage medium include:

is configured to perform the steps of determining a target loss function in the following manner:

는 타깃 손실 함수를 나타내고,

는 제1 훈련될 이미지 후보 영역을 나타내며,

는 제1 훈련될 텍스트를 나타내고,

는 제2 훈련될 이미지 후보 영역을 나타내며,

는 제2 훈련될 텍스트를 나타내고,

는 훈련될 데이터 쌍을 나타내며, max()는 최대값을 취함을 나타내고,

은 제1 파라미터 제어 가중치를 나타내며,

는 제2 파라미터 제어 가중치를 나타내고,

은 제1 미리 설정된 임계값을 나타내며,

는 제2 미리 설정된 임계값을 나타낸다.here,

represents the target loss function,

represents the first image candidate region to be trained,

represents the first to-be-trained text,

denotes the second to-be-trained image candidate region,

represents the second to-be-trained text,

denotes the data pair to be trained, max() denotes to take the maximum,

represents the first parameter control weight,

represents the second parameter control weight,

represents the first preset threshold,

denotes a second preset threshold value.

Embodiments of the present disclosure further provide a computer-readable storage medium comprising instructions, which, when executed on a computer, cause the computer to perform any possible implementation of the method for locating an image region provided in the above-described embodiments. or to perform any possible implementation of the model training method provided in the foregoing embodiments.

The above-described embodiment is only for explaining the technical solution of the present disclosure, not for limiting the present disclosure. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art may still add modifications or some technical solutions to the technical solutions described in the above-described embodiments without departing from the spirit and scope of the technical solutions of the embodiments of the present disclosure. It should be understood that features can be substituted equally.

Claims (20)

A method for finding an image area, comprising:
generating a set of region semantic information according to the set of image candidate regions of the to-be located image, wherein each region semantic information of the set of region semantic information is a set of image candidate regions. Corresponding to one image candidate area;
obtaining an enhanced semantic information set corresponding to the regional semantic information set using a Graph Convolutional Network (GCN), wherein each enhanced semantic information of the enhanced semantic information set is one of the regional semantic information sets. corresponding to the domain semantic information of , wherein the GCN is configured to establish an association relationship between various domain semantic information;
obtaining a degree of matching between the respective enhanced semantic information and a set of text features corresponding to the text to be found using an image region finding network model, wherein the image region finding network model is configured between the image candidate region and the text to be found. determine a matching relationship of , wherein each word of the text to be found corresponds to a single word feature of the set of text features; and
determining a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each of the enhanced semantic information;
A method for finding an image area containing According to claim 1,
The step of generating a region semantic information set according to the image candidate region set of the image to be found includes:
obtaining region semantic information corresponding to each image candidate region using a convolutional neural network (CNN), wherein the image candidate region includes region information, and the region information is an image of the image to be found including location information of the candidate area and size information of the image candidate area; and
generating the region semantic information set according to the N region semantic information when region semantic information corresponding to N image candidate regions is obtained, wherein N is an integer greater than or equal to 1
A method for finding an image region, comprising: 3. The method of claim 1 or 2,
Using the GCN to obtain an enhanced semantic information set corresponding to the regional semantic information set includes:
obtaining first region semantic information and second region semantic information from the region semantic information set, wherein the first region semantic information is any one region semantic information of the region semantic information set, and the second region semantic information is any one region semantic information of the region semantic information set;
obtaining a strength of a connection edge between the first region semantic information and the second region semantic information;
normalizing the strength of a connecting edge between the first region semantic information and the second region semantic information to obtain a normalized strength;
determining a target connection matrix according to normalized strengths between various region semantic information of the region semantic information set; and
determining an enhanced semantic information set corresponding to the target connection matrix using the GCN;
A method for finding an image region, comprising: 4. The method of claim 3,
Determining a target connection matrix according to the normalized strength between various region semantic information of the region semantic information set includes:
generating a connection matrix according to normalized strengths between various region semantic information of the region semantic information set; and
generating the target connection matrix according to the connection matrix and the unit matrix;
A method for finding an image region, comprising: 5. The method of claim 3 or 4,
Using the GCN to determine the enhanced semantic information set corresponding to the target connection matrix comprises:
the following way

calculating the enhanced semantic information as
here

represents the i-th enhanced semantic information corresponding to the k-th layer of GCN,

represents the j-th enhanced semantic information corresponding to the k-1 th layer of GCN,

represents the first network parameter of the k-th layer of GCN,

represents the second network parameter of the k-th layer of GCN,

indicates that the j-th node is a neighbor of the i-th node,

represents an element of the target connection matrix,
A method for finding an image area. 6. The method according to any one of claims 1 to 5,
Before using the image region finding network model to obtain a degree of matching between a set of text features corresponding to the text to be found and the respective enhanced semantic information, the method for finding the image region comprises:
obtaining the text to be found;
obtaining a text vector sequence according to the text to be found, the text vector sequence comprising T word vectors, each word vector corresponding to one word, and T being an integer greater than or equal to one;
encoding each word vector of the text vector sequence to obtain a text feature; and
generating the text feature set according to the T text features when text features corresponding to the T word vectors are obtained;
A method for finding an image area that further contains 7. The method of claim 6,
encoding each word vector of the sequence of text vectors to obtain the text feature comprises:
the following way

obtaining the text feature with
here

denotes the t-th text feature of the set of text features,

indicates that encoding is performed using a Long Short-Term Memory (LSTM) network,

represents the t-th word vector of the text vector sequence,

represents the (t-1)-th text feature of the text feature set,
A method for finding an image area. A model training method comprising:
obtaining a set of text to be trained and a set of image candidate regions to be trained, wherein the set of text to be trained includes a first text to be trained and a second text to be trained, and the set of image candidate regions to be trained is a first image to be trained image. a candidate region and a second candidate image to be trained region, wherein the first to-be-trained text and the first to-be-trained image candidate region have a matching relationship, and the first trained text and the second to-be-trained image candidate region does not have a matching relationship, the second to-be-trained text and the second to-be-trained image candidate region have a matching relationship, and the second to-be-trained text and the first to-be-trained image candidate region have a matching relationship. do not have ―;
determining a target loss function according to the first to-be-trained text, the second to-be-trained text, the first to-be-trained image candidate region, and the second to-be-trained image candidate region; and
training an image region finding network model to be trained using the target loss function to obtain an image region finding network model, wherein the image region finding network model is to be found with image candidate regions according to a text feature set and enhanced semantic information. determine a matching relationship between text, wherein the enhanced semantic information and the image candidate region have a correspondence, and the text feature set and the text to be found have a correspondence;
A model training method comprising 9. The method of claim 8,
determining a target loss function according to the first to-be-trained text, the second to-be-trained text, the first to-be-trained image candidate region, and the second to-be-trained image candidate region,
the following way

determining the target loss function as
here

represents the target loss function,

represents the first to-be-trained image candidate region,

represents the first to-be-trained text,

represents the second to-be-trained image candidate region,

represents the second to-be-trained text,

denotes the data pair to be trained, max() denotes to take the maximum,

represents the first parameter control weight,

represents the second parameter control weight,

represents the first preset threshold,

represents the second preset threshold,
How to train a model. A device for finding an image area, comprising:
a generating module, configured to generate a set of region semantic information according to a set of image candidate regions of an image to be found, wherein each region semantic information of the set of region semantic information corresponds to one image candidate region of the set of image candidate regions;
an obtaining module, configured to obtain, using a graph convolution network (GCN), an enhanced semantic information set corresponding to the regional semantic information set generated by the generating module, wherein each enhanced semantic information of the enhanced semantic information set is Corresponding to one region semantic information of a region semantic information set, the GCN is configured to establish an association relationship between various region semantic information,
The obtaining module is further configured to obtain a matching degree between the text feature set corresponding to the text to be found using the image area finding network model and the respective enhanced semantic information, wherein the image area finding network model is configured to: determine a matching relationship between a candidate region and the text to be found, wherein each word of the text to be found corresponds to one word feature of the set of text features; and
a determining module, configured to determine a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each enhanced semantic information obtained by the acquiring module
A device for finding an image area containing A model training device comprising:
an acquiring module configured to acquire a set of text to be trained and a set of image candidate regions to be trained, wherein the set of text to be trained includes a first text to be trained and a second text to be trained, and wherein the set of image candidate regions to be trained is a first training an image candidate region to be trained and a second image candidate region to be trained, wherein the first to-be-trained text and the first to-be-trained image candidate region have a matching relationship, and the first trained text and the second to-be-trained region The image candidate region does not have a matching relationship, the second to-be-trained text and the second to-be-trained image candidate region have a matching relationship, and the second to-be-trained text and the first to-be-trained image candidate region match not in a relationship ―;
a determining module, configured to determine a target loss function according to the first to-be-trained text, the second to-be-trained text, the first to-be-trained image candidate region, and the second to-be-trained image candidate region obtained by the acquiring module ; and
a training module, configured to train an image region finding network model to be trained using the target loss function determined by the determining module to obtain an image region finding network model, wherein the image region finding network model includes a text feature set and enhanced semantics and determine a matching relationship between the image candidate region and the text to be found according to the information, wherein the enhanced semantic information and the image candidate region have a correspondence, and the text feature set and the text to be found have a correspondence. —
A model training device comprising a. A terminal device comprising:
Memory, transceiver, processor and bus systems
includes,
The memory is configured to store a program,
The processor is
generating a set of region semantic information according to a set of image candidate regions of an image to be found, wherein each region semantic information of the set of region semantic information corresponds to one image candidate region of the set of image candidate regions;
obtaining an enhanced semantic information set corresponding to the set of region semantic information using a graph convolution network (GCN), wherein each enhanced semantic information of the set of region semantic information is one region semantic information of the set of region semantic information corresponding to , wherein the GCN is configured to establish an association relationship between various domain semantic information;
obtaining a degree of matching between the respective enhanced semantic information and a set of text features corresponding to the text to be found using an image region finding network model, wherein the image region finding network model is between the image candidate region and the text to be found. determine a matching relationship of , wherein each word of the text to be found corresponds to a single word feature of the set of text features; and
determining a target image candidate region from the image candidate region set according to a degree of matching between the text feature set and each of the enhanced semantic information;
configured to execute a program in the memory to perform
wherein the bus system is configured to couple the memory and the processor to enable the memory and the processor to communicate.
terminal device. 13. The method of claim 12,
The processor is
an operation of obtaining region semantic information corresponding to each image candidate region using a convolutional neural network (CNN), wherein the image candidate region includes region information, the region information comprising a location of an image candidate region of the image to be found information and size information of the image candidate region; and
An operation of generating the region semantic information set according to the N regions semantic information when region semantic information corresponding to N image candidate regions is obtained, wherein N is an integer greater than or equal to 1
further configured to execute the program in the memory to perform
terminal device. 14. The method of claim 12 or 13,
The processor is
obtaining first region semantic information and second region semantic information from the region semantic information set, wherein the first region semantic information is any one region semantic information of the region semantic information set, and the second region semantic information is any one region semantic information of the region semantic information set;
obtaining a strength of a connection edge between the first region semantic information and the second region semantic information;
normalizing the strength of a connecting edge between the first region semantic information and the second region semantic information to obtain a normalized strength;
determining a target connection matrix according to normalized strengths between various region semantic information of the region semantic information set; and
Determining an enhanced semantic information set corresponding to the target connection matrix using the GCN
further configured to execute the program in the memory to perform
terminal device. 15. The method of claim 14,
The processor is
generating a connection matrix according to normalized strengths between various region semantic information of the region semantic information set; and
generating the target connection matrix according to the connection matrix and the unit matrix
further configured to execute the program in the memory to perform
terminal device. 16. The method of claim 14 or 15,
The processor is
the following way

further configured to execute the program in the memory to perform the operation of calculating the enhanced semantic information with
here

represents the i-th enhanced semantic information corresponding to the k-th layer of GCN,

represents the j-th enhanced semantic information corresponding to the k-1 th layer of GCN,

represents the first network parameter of the k-th layer of GCN,

represents the second network parameter of the k-th layer of GCN,

indicates that the j-th node is a neighbor of the i-th node,

represents an element of the target connection matrix,
terminal device. As a server,
including memory, transceiver, processor and bus systems;
The memory is configured to store a program,
The processor is
generating a set of region semantic information according to a set of image candidate regions of an image to be found, wherein each region semantic information of the set of region semantic information corresponds to one image candidate region of the set of image candidate regions;
obtaining an enhanced semantic information set corresponding to the set of region semantic information using a graph convolution network (GCN), wherein each enhanced semantic information of the set of region semantic information is one region semantic information of the set of region semantic information corresponding to , wherein the GCN is configured to establish an association relationship between various domain semantic information;
obtaining a degree of matching between the respective enhanced semantic information and a set of text features corresponding to the text to be found using an image region finding network model, wherein the image region finding network model is between the image candidate region and the text to be found. determine a matching relationship of , wherein each word of the text to be found corresponds to a single word feature of the set of text features; and
determining a target image candidate region from the image candidate region set according to a degree of matching between the text feature set and each of the enhanced semantic information;
configured to execute a program in the memory to perform
wherein the bus system is configured to couple the memory and the processor to enable the memory and the processor to communicate.
server. As a server,
including memory, transceiver, processor and bus systems;
The memory is configured to store a program,
The processor is
obtaining a set of text to be trained and a set of image candidate regions to be trained, wherein the set of text to be trained includes a first text to be trained and a second text to be trained, and wherein the set of image candidate regions to be trained is a first image to be trained image. a candidate region and a second candidate image to be trained region, wherein the first to-be-trained text and the first to-be-trained image candidate region have a matching relationship, and the first trained text and the second to-be-trained image candidate region does not have a matching relationship, the second to-be-trained text and the second to-be-trained image candidate region have a matching relationship, and the second to-be-trained text and the first to-be-trained image candidate region have a matching relationship. do not have ―;
determining a target loss function according to the first to-be-trained text, the second to-be-trained text, the first to-be-trained image candidate region, and the second to-be-trained image candidate region; and
training an image region finding network model to be trained using the target loss function to obtain an image region finding network model, wherein the image region finding network model is to be found with image candidate regions according to a text feature set and enhanced semantic information. determine a matching relationship between text, wherein the enhanced semantic information and the image candidate region have a correspondence, and the text feature set and the text to be found have a correspondence;
configured to execute a program in the memory to perform
wherein the bus system is configured to couple the memory and the processor to enable the memory and the processor to communicate.
server. A method for finding an image area, comprising:
receiving an image finding command;
obtaining an image candidate region set of an image to be found according to the image finding command in response to the image finding command, wherein the image candidate region set includes N image candidate regions, where N is an integer greater than or equal to one; ;
generating a set of region semantic information according to the set of image candidate regions, wherein the set of region semantic information includes N pieces of region semantic information, each region semantic information corresponding to one image candidate region;
obtaining an enhanced semantic information set corresponding to the regional semantic information set using a graph convolutional network (GCN), wherein the enhanced semantic information set includes N enhanced semantic information, each enhanced semantic information comprising one corresponding to region semantic information, wherein the GCN is configured to establish an association relationship between various region semantic information;
obtaining a set of text features corresponding to the text to be looked up, wherein the text to be found comprises T words, the set of text features comprises T word features, each word corresponding to one word feature, and , T is an integer greater than or equal to 1 —;
obtaining a degree of matching between the set of text features and each of the enhanced semantic information using an image region finding network model, wherein the image region finding network model determines a matching relationship between the image candidate region and the text to be found configured to ―;
determining a target image candidate region from the image candidate region set according to a matching degree between the text feature set and each of the enhanced semantic information; and
sending the image creation command to the client so that the client can display the target image candidate region according to the image creation command
A method for finding an image area containing A computer readable storage medium comprising:
save the command,
The instructions, when executed on a computer, cause the computer to perform a method for finding an image region according to any one of claims 1 to 7 or a method for training a model according to claims 8 or 9. to do,
computer readable storage medium.

Images