JP2017143425A - Image feature descriptor encoder, image feature descriptor decoder, image feature descriptor encoding method and image feature descriptor decoding method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image feature descriptor encoder that improves prediction accuracy by using correlation in a time direction and achieves high encoding efficiency.SOLUTION: An encoder comprises: predicting means for setting an encoded descriptor signal on a reference picture, which a motion vector points, as a prediction descriptor signal for each feature point in a block; conversion means for generating a conversion signal for a difference value between the descriptor signal and the prediction descriptor signal; and encoding means for encoding a prediction mode, a conversion mode and conversion signals of respective feature points and generating a bit stream on which encoded data of the descriptor presence flag, the prediction mode, the conversion mode and the conversion signals of the respective feature points are multiplexed. The predicting means selects a prediction vector from one or more encoded motion vectors at the periphery of the block and generates a difference vector being a difference between the motion vector and the prediction vector. The encoding means encodes selection information of the prediction vector for each feature point and the difference vector, and multiplexes them on the bit stream.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an image feature descriptor encoding apparatus and method for encoding an image feature descriptor, and an image feature descriptor decoding apparatus and method for decoding an encoded feature descriptor.

  Image feature descriptor matching processing such as HOG (Histogram of Oriented Gradients), SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features) was used as a method for realizing image processing such as image search and object tracking. There is a technique. For example, when searching an image of a specific subject from a server connected via a network, the server stores feature descriptors of various images as a database, and searches by matching between feature descriptors with this database Suppose that In such an application, it is necessary to transmit the search target image itself and extract the feature descriptor of the search target image on the server side, or extract the feature descriptor on the user side and transmit the feature descriptor to the server. . Therefore, when the data size of the feature descriptor is smaller than the data size of the image itself, the transmission load can be reduced by transmitting the feature descriptor without transmitting the image itself.

  Non-Patent Document 1 is a technique developed for such a purpose, and shows a feature descriptor generation method and an encoding method in which the data size is smaller than the image itself. FIG. 1 shows a block diagram. In Non-Patent Document 1, first, an input image preprocessing unit 1 performs image reduction as preprocessing on an input image. Since the number of pixels of the reduced image, which is a preprocessed image, is smaller than that of the original input image, it is possible to reduce the calculation load and the number of feature points in the subsequent processing. Next, the feature descriptor generation unit 2 performs feature point extraction processing on the reduced image, and calculates a feature descriptor for the obtained feature point. Then, the descriptor descriptor corresponding to the obtained feature point position information is encoded as descriptor information by the descriptor information encoding unit 3 and output as an encoded bit stream. The control unit 4 controls the presence / absence of a process, selection of a processing method, and the like for each process.

ISO / IEC 15938-13: Compact descriptors for visual search

  Non-Patent Document 1 realizes transmission of a feature descriptor with a data size smaller than the image itself by the above processing. However, when handling a video composed of one or more images (frames) as an input image, Non-Patent Document 1 performs encoding using mainly the correlation in the frame for encoding processing, and thus the time inherent to the video There was a problem that direction correlation could not be used.

The present invention has been made to solve the above-described problems, and realizes high encoding efficiency by encoding processing using the time direction correlation of feature descriptors when encoding descriptor information. An object is to obtain an image feature descriptor encoding apparatus and an image feature descriptor encoding method.
The present invention also provides an image feature descriptor decoding device and an image feature description that can correctly decode the encoded bitstream generated by the image feature descriptor encoding device and the image feature descriptor encoding method as described above. An object is to obtain a child decoding method.

An image feature descriptor encoding device according to the present invention includes:
Block dividing means for dividing the input image into blocks;
In a block in which a descriptor presence / absence flag indicating that one or more descriptors are included in the block is true, when the prediction mode of the block is inter prediction, a motion vector is obtained for each feature point in the block. Prediction means that uses a coded descriptor signal on a reference picture indicated by
Conversion means for performing a conversion process on the difference value between the descriptor signal and the predicted descriptor signal according to the conversion mode of the block to generate a conversion signal;
Encode the descriptor presence flag, the prediction mode, the conversion mode, and the conversion signal of each feature point, and encode the descriptor presence flag, the prediction mode, the conversion mode, and the conversion signal of each feature point. Encoding means for generating a bitstream in which data is multiplexed;
With
The prediction means selects a prediction vector from one or more encoded motion vectors around the block that is a prediction vector candidate for each feature point, and a difference vector that is a difference between the motion vector and the prediction vector Produces
The encoding means encodes the prediction vector selection information and the difference vector for each feature point, and multiplexes the prediction vector selection information and the difference vector encoded data into a bitstream. To do.

  According to the present invention, the descriptor information encoding means encodes the prediction difference information by performing motion compensation prediction on the feature descriptor, and also predicts motion vectors used for motion compensation prediction from surrounding vectors. Since the difference vector obtained is encoded, the prediction accuracy is improved by using the correlation in the time direction, and high encoding efficiency can be realized.

It is a block diagram which shows the image feature descriptor encoding apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the example of the characteristic descriptor by Embodiment 1 of this invention. It is explanatory drawing which shows an example of the encoding order of the descriptor information by Embodiment 1 of this invention. It is explanatory drawing which shows an example of the encoding order of the descriptor information by Embodiment 1 of this invention. It is explanatory drawing which shows an example of the encoding order of the descriptor information by Embodiment 1 of this invention. It is a block diagram which shows the descriptor information encoding part by Embodiment 1 of this invention. It is explanatory drawing which shows the raster scan which is an example of the encoding order of the block unit of the image feature descriptor encoding apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the zigzag scan which is an example of the encoding order of the block unit of the image feature descriptor encoding apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the diagonal scan which is an example of the encoding order of the block unit of the image feature descriptor encoding apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the spiral scan which is an example of the encoding order of the block unit of the image feature descriptor encoding apparatus by Embodiment 1 of this invention. It is explanatory drawing which shows the quadtree scan which is an example of the encoding order of the block unit of the image feature descriptor encoding apparatus by Embodiment 1 of this invention. It is another block diagram which shows the descriptor information encoding part by Embodiment 1 of this invention. It is a block diagram which shows the image feature descriptor decoding apparatus by Embodiment 1 of this invention. It is another block diagram which shows the image feature descriptor decoding apparatus by Embodiment 1 of this invention. It is a flowchart which shows the processing content (image feature descriptor encoding method) of the image feature descriptor encoding apparatus by Embodiment 1 of this invention. 16 is a flowchart showing the processing content of “descriptor information encoding” in FIG. 15. FIG. 17 is a flowchart showing the processing content of “encoding descriptor information of encoding target pixel” in FIG. 16. FIG. It is explanatory drawing which shows the position of the prediction vector candidate by Embodiment 1 of this invention. It is a flowchart which shows the processing content (image feature descriptor decoding method) of the image feature descriptor decoding apparatus by Embodiment 1 of this invention. FIG. 20 is a flowchart showing the processing contents of “decoding descriptor information of decoding target pixels” in FIG. 19. FIG. It is explanatory drawing which shows the position of the prediction vector candidate by Embodiment 2 of this invention.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing an image feature descriptor encoding apparatus according to Embodiment 1 of the present invention. At the level of the block diagram, the configuration is the same as the conventional one, but the function of each element is different from the conventional one as will be described later.
The video signal to be processed by the image feature descriptor encoding apparatus according to the first embodiment is an arbitrary color space such as a YUV signal including a luminance signal and two color difference signals, or an RGB signal output from a digital image sensor. In addition to the above color video signal, the video frame is an arbitrary video signal such as a monochrome image signal or an infrared image signal, which is composed of a horizontal / vertical two-dimensional digital sample (pixel) sequence.
The gradation of each pixel may be 8 bits, or a gradation of 10 bits, 12 bits, or the like.
Further, it is natural that the input signal may be a still image signal instead of a video signal because the still image signal can be interpreted as a video signal composed of only one frame.

  In the following description, for the sake of convenience, unless otherwise specified, it is assumed that an input video signal is a YUV 4: 4: 4 format signal in which the two color difference components U and V are the same number of samples as the luminance component Y. . However, other 4: 4: 4 formats such as an RGB format composed of signals of three primary colors of red (R), green (G), and blue (B), and an HSV format composed of hue, saturation, and lightness may be used. . Alternatively, the YUV 4: 2: 0 format or the YUV 4: 2: 2 format may be used. The video format is converted from the format of the video signal input by the input image preprocessing unit 1 into a format for generating an image feature descriptor (if necessary).

  It should be noted that the format for generating the image feature descriptor may be encoded with the upper header as index information so that the image feature descriptor decoding apparatus can recognize it. In the case of a 4: 4: 4 format signal, each color signal may be regarded as a monochrome image signal (YUV4: 0: 0) and encoded independently to generate a bitstream. At this time, information indicating which color signal each monochrome signal is may be encoded as index information with an upper header such as a sequence level header described later so that the image feature descriptor decoding apparatus can recognize it. By doing so, it is possible to identify the target color signal of the decoded feature descriptor decoded by the image feature descriptor decoding apparatus.

In the above description, the case where each color signal of the 4: 4: 4 format signal is regarded as a monochrome image signal and independently encoded is described. However, a monochrome (YUV4: 0: 0) code is actually used for a monochrome image signal. Naturally, it is also possible.
In the above description, the YUV signal format and the RGB signal format have been described. However, other color signal formats (YCbCr, XYZ, etc.) are similarly 4: 2: 0, 4: 2: 2, 4: 4. : Any of the four formats can be encoded in the same manner as the YUV format. However, it is not limited (which can be arbitrarily set) which color signal in the target format corresponds to which color signal in the YUV format. Of course, as described above in the case of the RGB format, the color signal association may be encoded as index information in the upper header so that the image feature descriptor decoding apparatus can recognize it.

  The processing data unit corresponding to each frame of the video is referred to as “picture”. In the first embodiment, “picture” is described as a signal of a video frame that has been sequentially scanned (progressive scan). However, when the video signal is an interlace signal, the “picture” may be a field image signal which is a unit constituting a video frame.

  In FIG. 1, an input image preprocessing unit 1 inputs an input image signal and performs preprocessing. Preprocessing includes filter processing such as resolution conversion, color format conversion, and noise removal processing. At that time, a resolution conversion instruction, a color format conversion instruction, and a filter processing instruction are issued from the control unit 4. For example, as in Non-Patent Document 1, the control unit 4 instructs a fixed process such as reducing the input image in a vertical size and a horizontal direction with a preset size and a set reduction process, respectively. Alternatively, the control unit 4 dynamically issues a resolution conversion instruction, a resolution conversion filter and conversion size instruction, a filter processing instruction, a filter to be used, and the like according to a predetermined rule.

The feature descriptor generation unit 2 extracts feature points from the preprocessed image signal input from the input image preprocessing unit 1, and generates a feature descriptor for each feature point.
An example of feature point extraction will be given below. First, a large number of filter processed images used when extracting feature points are generated. Specifically, Gaussian filters having different smoothing intensities are prepared, and a smoothed image by each filter is generated as one of the filtered images. Further, an image obtained by further down-sampling the smoothed image is also generated as one of the filtered images.

  Next, FIG. 2 shows an example of feature descriptor generation. FIG. 2 uses a 128-dimensional vector as a feature descriptor for a certain feature point. Specifically, the gradient direction of the feature points is first calculated for the preprocessed image signal. For a preprocessed image signal rotated so that the gradient direction is a horizontal direction (or any direction as long as it is a fixed direction such as a vertical direction), a predetermined surrounding area centered on a feature point is obtained. Divide into 16 square blocks. Then, the gradient vector of each pixel is quantized in eight directions for each block and the magnitude is added. 2 indicate the total sum of the magnitudes of gradient vectors (quantized in 8 directions) in the block in each direction. The above processing is performed on each block to generate 128-dimensional vector information, which is used as a feature descriptor for the feature point.

Further, the feature descriptor generation unit 2 performs a process of selecting the extracted feature points based on the control instruction of the control unit 4. For example, when the control unit 4 controls the transmission bit rate (or data size of the encoded bit stream) of the encoded bit stream output from the descriptor information encoding unit 3, the descriptor information encoding unit 3 described later encodes The number of feature points is controlled so that the bit rate of the encoded bit stream obtained by encoding the descriptor information is equal to or less than the bit rate of the control target. Alternatively, the number of feature points per picture is defined by a picture level header to be described later, and feature descriptors corresponding to the number of feature points are selected.
The feature descriptor generation unit 2 outputs the extracted feature point position information and the generated feature descriptor to the descriptor information encoding unit 3 as descriptor information.

The descriptor information encoding unit 3 encodes the descriptor information of each picture input from the feature descriptor generation unit 2 to generate an encoded bit stream and outputs it.
Here, examples of the coding order of pictures are shown in FIGS. 3 and 4 show an encoding structure having the same encoding order and display order (time order), and FIG. 5 shows an encoding structure in which the encoding order is changed to an order different from the display order.

  Further, these coding structures also define a reference structure for prediction processing described later. FIG. 3 shows an encoding structure that uses only an I (Intra) picture (a picture in which a prediction mode described later is a direct encoding mode only) that does not use prediction that refers to an encoded picture. The bit stream can be decoded from any picture. Next, FIG. 4 shows a code that uses a P (Predictive) picture (a picture in which a prediction mode to be described later becomes a direct encoding mode or an inter prediction mode) that enables prediction referring to an encoded picture in addition to an I picture. The encoded bit stream generated with this structure can be decoded only from the I picture. Inter prediction in a P picture refers to one reference picture from one or more encoded pictures shown in a reference picture list described later. FIG. 5 also uses a B (Bi-predictive) picture that enables prediction with reference to one or two reference pictures from one or more encoded pictures shown in the reference picture list in addition to the I picture and P picture. It is a coding structure. In this structure, the coding delay is caused by changing the coding order from the display order, but the coding efficiency by the B picture can be improved.

  The image feature descriptor decoding apparatus decodes picture type information (information indicating any one of an I picture, a P picture, and a B picture) whose encoding order is a part of header information (picture level header) of the encoded bitstream. Thus, the reference candidate relationship for prediction of each picture can be determined by decoding a reference picture list described later.

FIG. 6 shows details of the descriptor information encoding unit 3. Regarding the arrows in FIG. 6, the black circles indicate the same arrow technique. On the other hand, the intersections of arrows without black circles are independent of each other.
The dividing unit 101 sets descriptor position information corresponding to the descriptor signal, which is the descriptor information extracted in the picture, in a predetermined order with respect to the encoding target picture, in a predetermined order, the changeover switch 102, the descriptor information memory 104, Processing to be output to the encoding unit 107 is performed. Specifically, the descriptor signal is output to the changeover switch 102, and the descriptor position information is output to the descriptor information memory 104 and the encoding unit 107.

  The predetermined order includes a predetermined order in block units and a predetermined order in pixel units. First, a picture is divided into blocks and processed according to a predetermined order in block units. Each block has a descriptor presence / absence flag, and indicates true (1) when there is one or more descriptor information in the block, and false (0) when none exists. Then, for only a block whose descriptor presence / absence flag is true, each descriptor information in the block is processed according to a predetermined order in pixel units such as raster scan. Examples of the predetermined order in block units are shown in FIGS. 7 to 11, each is divided into reference blocks and processed in units of reference blocks. The arrows in each figure indicate the processing order of the reference block. 7 is a raster scan processed horizontally from the upper left of the picture, FIG. 8 is a zigzag scan processed zigzag from the upper left of the picture, FIG. 9 is an oblique scan processed diagonally from the lower left, and FIG. 10 is the center of the picture Shows spiral scan (circle scan) processed clockwise.

  FIG. 11 shows a quadtree scan. First, similarly to FIGS. 7 to 10, a descriptor presence / absence flag is provided for each reference block. Further, the block having the descriptor presence / absence flag true (1) is further divided into four blocks. Each of the four divided blocks has a descriptor presence / absence flag, and a block whose descriptor presence / absence flag is true (1) is further divided into four blocks. For blocks without descriptors, the above-mentioned descriptor presence / absence flag is set to false (0) and no further division is performed. This quadtree partitioning is hierarchically divided up to the set minimum block size, and processing in units of pixels is performed on the minimum size block for which the descriptor presence / absence flag is true.

The processing in units of pixels in the block is to process the descriptor presence / absence flags in units of pixels in a predetermined order such as raster scanning.
The descriptor presence / absence flags in block units and pixel units indicate descriptor position information.
Further, the dividing unit 101 performs a process of separating and outputting the descriptor information into the descriptor signal and the descriptor position information.

  The changeover switch 102 refers to the prediction mode information input from the control unit 4 and switches the changeover switch. Specifically, when the prediction mode information indicates the inter prediction mode, the descriptor signal is output to the subtraction unit 105 and the descriptor information memory 104. On the other hand, when the prediction mode indicates the direct encoding mode, the prediction mode is output to the direct conversion unit without being output to the subtraction unit 105.

  The prediction unit 103 performs inter prediction with reference to the descriptor information memory 104. That is, the descriptor to be encoded is searched for a descriptor to be used for prediction from descriptors of reference pictures (encoded pictures that can be referred to by the prediction unit 103), and the descriptor detected as a result of the search is used as a prediction descriptor signal. As shown in FIG. The motion vector indicating the position of the prediction descriptor (position information of the prediction descriptor starting from the position of the encoding target descriptor) is encoded as part of inter prediction information as a difference vector from the prediction vector described later. 107 is encoded. The inter prediction information includes reference picture information for specifying a reference picture in which a prediction descriptor exists, prediction vector information obtained from an encoded surrounding motion vector, and a difference vector between the motion vector and the prediction vector. Is done.

As a search method, for example, when the encoding target descriptor is the 128-dimensional vector shown in FIG. 2, a method that uses a prediction descriptor signal that minimizes the sum of absolute values of the differences between the vectors, There is a technique in which a signal with the smallest sum of squared differences is used as a prediction descriptor signal.
A prediction descriptor signal may be generated from a plurality of descriptors. Specifically, two descriptors are selected from one or more reference pictures, and their weighted average value is used as a prediction descriptor signal. A combination of two descriptors may be selected from different pictures, or descriptors at different positions of the same reference picture may be selected. By using the weighted average value of the two descriptors as the prediction descriptor signal, the noise reduction effect of the prediction value can be obtained and the prediction accuracy can be improved. Note that information on whether a prediction descriptor signal is generated from a plurality of descriptors or a single descriptor is encoded as part of inter prediction information.

  The descriptor information memory 104 performs processing for storing the descriptor picture signal of the reference picture and the corresponding descriptor position information. The reference picture descriptor signal and descriptor position information are collectively referred to as reference descriptor information. The reference picture is controlled by the control unit 4. The control method may be a fixed method such as always using the immediately preceding picture in the encoding order, or may be dynamically controlled on a picture-by-picture basis. For example, picture numbers that are reference candidates are managed as a reference picture list and updated in units of pictures. The update information of the list is encoded by the encoding unit 107 and multiplexed into the encoded bitstream, so that the same reference picture list can be generated by decoding the encoded bitstream on the image feature descriptor decoding side. it can.

  The update information of the reference picture list can be represented by a difference value of the picture number from the reference picture list at the time of encoding the previous picture in the code order. The number of reference pictures may be variable for each picture. In that case, a new picture number is encoded. The number of reference pictures for each picture is encoded in sequence units or picture units. The maximum number of reference pictures is encoded in sequence units. By defining the maximum number of reference pictures, the capacity of the descriptor information memory 104 required by the image feature descriptor encoding device (similarly, the descriptor information memory 204 required by the image feature descriptor decoding device) Capacity) can be defined. For example, when it is important to set the capacity of the descriptor information memory 104 to be lower than the prediction accuracy, the capacity required for the descriptor information memory 104 can be reduced by setting the maximum number of reference pictures to a small value.

  The subtracting unit 105 performs a process of subtracting the prediction descriptor signal output from the prediction unit 103 from the descriptor signal output from the changeover switch 102 and outputting the subtraction result to the conversion unit 106 as a difference descriptor signal. To do.

  The conversion unit 106 refers to the conversion mode determined by the control unit 4, performs conversion processing on the differential descriptor signal output from the subtraction unit 105, and outputs the conversion result to the encoding unit 107 as a conversion signal To implement. That is, one or more conversion methods are prepared, and these conversion methods are selected according to the conversion mode. For the conversion method, for example, the following conversion method is prepared for the eight components h0, h1,..., H7 belonging to one cell in the 128-dimensional vector of FIG. In this case, the selection information for each cell corresponds to the conversion mode. Or you may set so that it may carry out by the method decided beforehand for every cell. In such a configuration, the conversion mode becomes unnecessary. For example, a cell adjacent to a cell implementing Transform A performs Transform B, and a cell adjacent to a cell implementing Transform B is switched alternately like black and white on a checkerboard so as to implement Transform A. There is a way to make it.

Transform A
v0 = (h2-h6) / 2
v1 = (h3-h7) / 2
v2 = (h0-h1) / 2
v3 = (h2-h3) / 2
v4 = (h4-h5) / 2
v5 = (h6-h7) / 2
v6 = ((h0 + h4)-(h2 + h6)) / 4
v7 = ((h0 + h2 + h4 + h6)-(h1 + h3 + h5 + h7)) / 8
Transform B
v0 = (h0-h4) / 2
v1 = (h1-h5) / 2
v2 = (h7-h0) / 2
v3 = (h1-h2) / 2
v4 = (h3-h4) / 2
v5 = (h5-h6) / 2
v6 = ((h1 + h5)-(h3 + h7)) / 4
v7 = ((h0 + h1 + h2 + h3)-(h4 + h5 + h6 + h7)) / 8

  Further, “no conversion” may be prepared as one of the conversion modes so that the presence or absence of conversion can be controlled. In this way, it is possible to selectively convert only a region where the power concentration degree due to conversion is high, and it is possible to suppress a decrease in coding efficiency due to conversion for a region where the power concentration degree due to conversion is low.

The encoding unit 107 outputs the conversion signal output from the conversion unit 106, the descriptor position information output from the division unit 101, the prediction mode and conversion mode output from the control unit 4, and the interface output from the prediction unit 103. A process of generating the encoded bitstream by encoding the prediction information as descriptor data is performed.
Examples of the encoding method for each parameter include entropy codes such as arithmetic codes and Huffman codes.
Also, the encoding unit 107 encodes header information of the encoded bit stream. As header information, a sequence level header and a picture level header are encoded, and a process of generating an encoded bit stream together with descriptor data is performed. The reference picture management information is encoded as a picture level header.

The sequence level header is a collection of header information that is generally common to sequence units, such as the image size, color signal format, and the maximum number of reference pictures during inter prediction.
The picture level header is a collection of header information set in units of pictures, such as an index of a sequence level header to be referred to, picture type information, an entropy coding probability table initialization flag, and the above reference picture list.
Also, it has the maximum number of feature points per picture as one of sequence level header or picture level header. This makes it possible to control the maximum number of feature points that a picture has. By reducing this parameter, it is possible to generally reduce the load of encoding and decoding processing and reduce the bit rate of the encoded bit stream output from the image feature descriptor encoding apparatus.
Furthermore, the picture level header has flag information (descriptor presence flag) indicating whether or not a feature descriptor (feature point) exists in the picture. In the image feature descriptor encoding apparatus, whether or not a descriptor exists can be determined by whether or not descriptor position information of the picture exists. The image feature descriptor decoding apparatus can identify whether or not a feature descriptor exists in the picture by decoding this flag from the encoded bit stream.

Each header information and later-described picture data are identified by a NAL (Network Abstraction) unit. Specifically, picture data in which a sequence level header, a picture level header, and descriptor data are grouped in units of pictures is defined as a unique NAL unit type, and is encoded together with identification information (index) of the NAL unit type. Supplemental information is also defined as a unique NAL unit if it exists. The access unit indicates a unit of data access of each picture in which NAL units related to the picture such as a NAL unit indicating picture data and a NAL unit indicating supplementary information are collected.
Further, the picture data has a picture level header index to be referred to as a header division method.

In the example of FIG. 1, each of the division unit 101, the changeover switch 102, the prediction unit 103, the subtraction unit 105, the conversion unit 106, and the encoding unit 107, which are components of the image feature descriptor encoding device, is dedicated hardware. Although a configuration (for example, a semiconductor integrated circuit mounted with a CPU or a one-chip microcomputer) is assumed, the image encoding device may be configured with a computer.
When the image feature descriptor encoding apparatus is configured by a computer, the descriptor information memory 104 is configured on the computer memory, and the dividing unit 101, the changeover switch 102, the prediction unit 103, the subtraction unit 105, the conversion unit 106, A program describing the processing contents of the encoding unit 107 may be stored in a memory of a computer, and the CPU of the computer may execute the program stored in the memory.

  FIG. 12 shows another configuration of the image feature descriptor encoding apparatus. In this configuration, the prediction unit 103 is changed to the prediction unit 108 as compared with the configuration of FIG. The prediction unit 108 receives a prediction mode from the control unit 4 and switches the prediction method according to the prediction mode. That is, it has a function equivalent to that of the combination of the changeover switch 102 and the prediction unit 103. Specifically, when the prediction mode indicates the inter prediction mode, inter prediction with reference to the descriptor information memory 104 is performed, and a prediction descriptor signal obtained by the inter prediction is output to the subtraction unit 105. The inter prediction information obtained in the inter prediction is output to the encoding unit 107. On the other hand, when the prediction mode indicates the direct encoding mode, the prediction descriptor signal is output to the subtraction unit 105 as a descriptor whose values are all 0. Further, since there is no inter prediction information in the case of the main prediction, the inter prediction information is not output to the encoding unit 107.

FIG. 13 is a block diagram showing an image feature descriptor decoding apparatus according to Embodiment 1 of the present invention.
In FIG. 13, when the decoding unit 201 inputs the encoded bit stream generated by the image feature descriptor encoding apparatus of FIG. 1, each header information and description such as a sequence level header and a picture level header are described from the bit stream. A process of decoding the child data is performed.

  At this time, in the sequence level header, each signal of the YUV 4: 4: 4 format signal and the RGB 4: 4: 4 format signal is regarded as a monochrome image signal and is independently encoded in monochrome (YUV4: 0: 0). When the information to be shown is included as header information, decoding processing can be performed independently on the encoded bit stream of each color signal.

  Each header information and picture data is identified by a NAL unit. Specifically, the sequence level header, the picture level header, and the picture data are each defined as a unique NAL unit type, and are identified by decoding identification information (index) of the NAL unit type. The supplemental information is also identified as a unique NAL unit if it exists. Further, the access unit is identified as a unit in which NAL units related to the picture such as NAL units indicating picture data and supplementary information are collected.

  In addition, a picture level header referred to by the picture data is identified from the index of the picture level header existing in the header information of the picture data, and a sequence level header referenced from the index of the sequence level header existing in the picture level header is specified. .

The decoding unit 201 performs a process of decoding the reference picture management information in the picture level header and outputting it to the descriptor information memory 204.
Further, since the descriptor data of the picture exists only when the descriptor presence flag of the picture level header is true, the processing of each unit 202 to 206 is performed. That is, when the presence flag indicates false, the descriptor data of the picture does not exist, and the process proceeds to the decoding process of the next picture in the decoding order.

Also, the decoding unit 201 decodes the descriptor data to obtain a converted signal, descriptor position information, prediction mode, conversion mode, and inter prediction information. The conversion signal is output to the inverse conversion unit 202, the descriptor position information is output to the descriptor information memory 204, the prediction mode is output to the changeover switch 205, the conversion mode is output to the inverse conversion unit 202, and the inter prediction information is output to the prediction unit 203. The position information is output as an output of the image feature descriptor decoding apparatus. The motion vector is obtained by a decoding process in which a prediction vector obtained from the decoded motion vector and the decoded prediction vector information is used as a prediction value.
As described above, the descriptor position information is composed of the descriptor presence / absence flags in block units and pixel units, and is decoded by the decoding unit 201 in the same block unit and pixel unit processing order as the image feature descriptor encoding apparatus. The position of the descriptor is specified according to the descriptor position information.

  The inverse conversion unit 202 refers to the conversion mode decoded by the decoding unit 201, performs an inverse conversion process on the conversion signal decoded by the decoding unit 201, and is the same as the difference descriptor signal in FIG. A process for calculating a differential descriptor signal is performed.

  The prediction unit 203 performs an inter prediction process similar to that performed by the prediction unit 103 of the image feature descriptor encoding device. That is, a process of outputting the descriptor stored in the descriptor information memory 204 indicated by the inter prediction information decoded by the decoding unit 201 to the adding unit 206 as a prediction descriptor signal is performed.

  Based on the reference picture management information decoded by the decoding unit 201, the descriptor information memory 204 performs a process of managing the reference picture descriptor signal as reference descriptor information and the corresponding descriptor position information. . Specifically, whether to refer to the reference descriptor information of each reference picture is set according to the reference picture management information. Further, the reference picture stored in the descriptor information memory 204 is managed (recorded or deleted) in accordance with the reference picture management information and the maximum number of reference pictures at the time of inter prediction decoded as part of the sequence level header.

  The changeover switch 205 refers to the prediction mode decoded by the decoding unit 201 and switches the changeover switch. Specifically, when the prediction mode indicates the inter prediction mode, the difference descriptor signal is output to the addition unit 206. On the other hand, when the prediction mode indicates the direct encoding mode, it is not output to the adding unit 206 but is output as a descriptor signal that is an output of the image feature descriptor decoding apparatus.

  The addition unit 206 adds the prediction descriptor signal output from the changeover switch 205 and the prediction descriptor signal output from the prediction unit 203, and the addition result is a descriptor that is an output of the image feature descriptor decoding apparatus. Output as a signal.

In the example of FIG. 13, each of the decoding unit 201, the inverse conversion unit 202, the prediction unit 203, the descriptor information memory 204, the changeover switch 205, and the addition unit 206, which are components of the image feature descriptor decoding device, is dedicated hardware. It is assumed that the image decoding apparatus is configured by hardware (components other than the descriptor information memory 204 are configured by, for example, a semiconductor integrated circuit mounted with a CPU or a one-chip microcomputer) May be configured by a computer.
When the image feature descriptor decoding apparatus is configured by a computer, the descriptor information memory 204 is configured on the computer memory, and the decoding unit 201, the inverse conversion unit 202, the prediction unit 203, the changeover switch 205, and the addition unit 206 A program describing processing contents may be stored in a memory of a computer, and a CPU of the computer may execute the program stored in the memory.

  FIG. 14 shows another configuration of the image feature descriptor decoding apparatus. In this configuration, the prediction unit 203 is changed to the prediction unit 207 as compared with the configuration of FIG. The prediction unit 207 performs the same prediction process as the prediction unit 108 of the image feature descriptor encoding device. The prediction unit 207 receives the prediction mode decoded by the decoding unit 201, and switches the prediction method according to the prediction mode. That is, it has a function equivalent to that of the combination of the changeover switch 205 and the prediction unit 207. Specifically, when the prediction mode indicates the inter prediction mode, inter prediction with reference to the inter prediction information decoded by the decoding unit 201 and the descriptor information memory 204 is performed, and the prediction descriptor obtained by the inter prediction The signal is output to the adding unit 206. On the other hand, when the prediction mode indicates the direct encoding mode, the prediction descriptor signal is output to the adding unit 206 as a descriptor having all values of 0.

Next, the operation will be described.
FIG. 15 is a flowchart showing the processing contents (image feature descriptor encoding method) of the image feature descriptor encoding apparatus according to Embodiment 1 of the present invention.
In the first embodiment, each frame image of video is used as an input image, inter prediction using encoded pictures is performed, conversion processing is performed on the obtained prediction descriptor signal, and conversion after conversion is performed. An image feature descriptor encoding apparatus that encodes descriptor data including a signal and header information to generate an encoded bit stream, and an encoded bit stream output from the image feature descriptor encoding apparatus An image feature descriptor decoding apparatus will be described.

The image feature descriptor encoding apparatus in FIG. 1 is characterized in that a feature point and a feature descriptor at that point are generated from a video signal, and inter-prediction encoding is performed in accordance with a change in the time direction of the feature point.
In general, video signals have a high temporal correlation, and feature descriptors tend to have a small temporal change and a small change in adjacent pictures. Therefore, highly accurate prediction can be realized by performing inter prediction. On the other hand, there are cases where the prediction efficiency of inter prediction is low due to poor correlation in the time direction, such as feature points changing / disappearing due to occlusion, or new feature points appearing due to the appearance of new objects. Therefore, it is desirable to reduce the power / entropy of the prediction difference signal by switching the presence / absence of inter prediction according to the change of the feature point in the time direction as described above. In the present embodiment, such an image feature descriptor encoding device that can adaptively switch the presence / absence of inter prediction is realized.

  In the first embodiment, in order to perform coding adapted to the general properties of such a video signal, a configuration is adopted in which presence / absence of inter prediction processing is adapted to each block divided according to predetermined block division processing. I am doing so.

First, the processing contents of the image feature descriptor encoding apparatus in FIG. 1 will be described.
First, whether a video signal is input to the input image preprocessing unit 1 of the image feature descriptor encoding apparatus (step ST1), and it is necessary to perform preprocessing on the input video signal based on a predetermined determination method A determination of whether or not is made (step ST2). When it is determined that preprocessing is necessary, preprocessing is performed on the video signal (step ST3). As an example of the determination of the presence / absence of preprocessing, it is possible to determine that reduction processing to a defined size is necessary as preprocessing if the resolution is equal to or larger than a predetermined size. If it is determined that preprocessing is necessary, reduction processing to the size defined as preprocessing is performed. Examples of other preprocessing include color format conversion and filter processing such as noise removal processing.

Next, the feature descriptor generator 2 extracts feature points for each picture from the converted video signal (step ST4). If it is determined in ST2 that preprocessing is not necessary, ST4 is performed on the video signal input in ST1. In the feature point extraction process, the extracted feature point is selected based on a control instruction from the control unit 4. For example, when the control unit 4 controls the transmission bit rate (or data size of the encoded bit stream) of the encoded bit stream output from the descriptor information encoding unit 3, the descriptor information encoding unit 3 encodes the encoded bit stream. The number of feature points is controlled so that the bit rate of the encoded bit stream obtained by encoding the descriptor information is equal to or lower than the target bit rate. Specifically, first, an evaluation function for calculating the importance of each feature point is defined, and an index p (p = 1, in order of feature points having the highest importance based on the score of each feature point indicated by the defined evaluation function. 2,..., P, P are assigned the number of feature points in the picture. Then, the prediction code amount Cp of each feature point is calculated. Then, PMax satisfying the following formula is calculated. PMax is the number of feature points of the picture.

ただし、Ｔは制御目標のビットレート及び符号化済みピクチャの符号量から算出される当該ピクチャでの制御目標符号量である。Ｔの算出例を下記に示す。

However, T is the control target code amount for the picture calculated from the bit rate of the control target and the code amount of the encoded picture. An example of calculating T is shown below.

ここで、Ｒは制御目標のビットレート、Ｆｒはフレームレート、Ｍは１秒毎にカウントが０となるピクチャ番号（Ｍ＝１、２、・・・、Ｆｒ）である。

Here, R is a control target bit rate, Fr is a frame rate, and M is a picture number (M = 1, 2,..., Fr) whose count is 0 every second.

Similarly, the feature descriptor generation unit 2 generates a feature descriptor for the feature point after the selection process (step ST5). Examples of feature points are as described above. The position information of the feature point and the generated feature descriptor are output to the descriptor information encoding unit 3 as descriptor information.
The descriptor information encoding unit 3 encodes the descriptor information based on the control instruction of the control unit 4 to generate an encoded bit stream (step ST6). A specific encoding process operation will be described next.

FIG. 16 is a flowchart showing the detailed processing contents of step ST6. In step ST6, the process shown in FIG. 16 is performed every time descriptor information in units of pictures is input.
First, the descriptor information memory 104 sets a reference picture to be used for inter prediction on a decoding target picture according to the control unit 4, and updates the reference picture list (step ST11). Next, the encoding unit 107 encodes the header information of the picture data of the picture to be encoded and the picture level header referred to by the header information of the picture data (step ST12). However, if the picture is the head of the sequence, the sequence level header is also encoded. In addition, when the picture level header referred to in the encoding target picture has already been encoded, the picture level header is not encoded.

  Next, it is determined whether or not one or more feature descriptors exist in the picture to be encoded (whether or not the descriptor presence flag is “1 (true)”) (step ST13). When the descriptor presence flag is “0 (false)”, the process of FIG. 16, that is, the process of step ST6 of FIG. When the descriptor presence flag is “1 (true)”, block division of the picture is performed by the method described with reference to FIGS. 7 to 11 described above (step ST14). However, in the case of FIG. 11, the process up to the division of the reference block is performed. Then, a reference block (encoding target block) that is the first encoding target is set (step ST15), and encoding processing in units of reference blocks is performed in the processing order of FIGS.

  First, it is confirmed whether or not one or more descriptors exist in the encoding target block (step ST16), and when one or more descriptors exist in the encoding target block, a block unit descriptor presence flag is set. "1 (with descriptor)" is encoded (step ST17). On the other hand, when there is no descriptor in the encoding target block, encoding “0 (no descriptor)” is encoded as the descriptor presence / absence flag for each block, thereby completing the encoding of the descriptor information of the pixel ( Step ST18).

If one or more descriptors exist in the encoding target block, the prediction mode and the transformation mode in the encoding target block are determined according to the control unit 4 (step 19). Then, the first encoding target pixel in the encoding target block is set (step ST20), and the descriptor information of the encoding target pixel is encoded (step ST21). Detailed processing of encoding the descriptor information of the encoding target pixel will be described later. Until the encoding process for the last pixel in the encoding target block is completed (step ST22), the descriptor information is encoded for each pixel in accordance with a predetermined processing order such as raster scan (step ST23).

After encoding the descriptor information of all the pixels in the encoding target block, until the encoding process for the last block in the encoding target picture is completed (step ST24), the reference block according to a predetermined processing order The descriptor information is encoded every time (step ST25).
However, in the case of the processing order of FIG. 11, the processing of step ST19 to step ST23 is performed only when the encoding target block is the minimum block. That is, the reference block is divided hierarchically until the encoding target block becomes the minimum block, and the descriptor presence / absence flag indicating whether one or more descriptors exist in the block in each block of each hierarchy is encoded. Shall.

FIG. 17 is a flowchart showing the detailed processing content of step ST21. First, it is determined whether or not descriptor information exists in the encoding target pixel, that is, whether or not the encoding target pixel is a feature point output from the feature descriptor generation unit 2 (step ST31). When the descriptor information exists in the encoding target pixel (the encoding target pixel is a feature point), “1 (with descriptor)” is encoded as a descriptor presence / absence flag for each pixel (step ST32). . On the other hand, when there is no descriptor information in the encoding target pixel (the encoding target pixel is not a feature point), “0 (no descriptor)” is encoded as a descriptor presence / absence flag for each pixel (step ST33).
Furthermore, when the descriptor information exists in the encoding target pixel, the prediction mode of the encoding target block to which the encoding target pixel belongs is referred to (step ST34). When the referred prediction mode is the inter prediction mode, the prediction unit 103 performs inter prediction of the pixel (step ST35). That is, the encoding target descriptor searches for a descriptor to be used for prediction from the descriptors of the reference picture, and performs a process of outputting the detected descriptor as a prediction descriptor signal to the subtraction unit 105. The search process at this time is performed based on the search range and the evaluation function set by the control unit 4. As an example of the evaluation function, when the encoding target descriptor is the 128-dimensional vector shown in FIG. 2, the absolute value sum of the difference of each vector or the square error sum of the difference of each vector can be cited. The smallest descriptor is the predicted descriptor signal.

  Next, a motion vector indicating the position of the determined prediction descriptor signal is subtracted from a prediction vector obtained from surrounding encoded motion vectors to obtain a difference vector (step ST36). Next, the prediction vector will be described in detail.

  FIGS. 18A to 18D show examples of prediction vector candidates. In each example, a prediction vector candidate is generated from a motion vector that has been encoded for each encoding target block to which the encoding target pixel belongs. In addition, each example shows an example in which the encoding target block is 4 × 4 pixels. FIG. 18A shows a pixel (A0) adjacent to the lower left, a pixel (B0) adjacent to the upper right, a pixel (B2) adjacent to the upper left, and an adjacent pixel above A0 with respect to the encoding target block. (A1), a motion vector of the pixel (C) of the previous picture in the encoding order at the position diagonally lower right from the center of the encoding target block (B1), one adjacent pixel to the left of B0 It is a candidate. However, for the pixels having no motion vector among the above-mentioned pixels, prediction vector candidates are set according to a predetermined procedure. As an example of the predetermined procedure, among the encoded motion vectors, a method of setting a motion vector having a spatial distance closest to a pixel where no motion vector exists, or a fixed vector such as a zero vector is set. There is a technique. Examples of the spatial distance include Euclidean distance and Manhattan distance. Further, C may be a position adjacent to the encoding target block at the lower right instead of the above position.

After setting the prediction candidate vector, a prediction vector is selected based on the evaluation function set by the control unit 4. As an example of the evaluation function, the absolute value sum | dx | + | dy | of each component when the difference vector between the motion vector to be encoded and the prediction vector candidate is (dx, dy) or the square sum dx of each component ² + dy ² is cited, and a prediction vector candidate that minimizes this evaluation function is set as a prediction vector.
Then, an index is set for each prediction vector candidate (for example, indexes are assigned in the order of A0, B0, B2, C, A1, and B1), and the index of the prediction vector candidate selected as the prediction vector is set in the prediction vector information.

  18 (b) to 18 (d) show a part obtained by removing some prediction vector candidates from FIG. 18 (a). As the number of prediction vector candidates decreases, the motion vector prediction accuracy decreases. However, since the maximum value of the index decreases, the amount of encoded data related to the prediction vector information can be reduced.

  In addition, as an example of prediction vector candidates other than the example of FIG. 18, there is a method in which the most recent M encoded motion vectors (M: an integer of 1 or more) in the encoding order are used as prediction vector candidates. In this case, since the strict spatial correlation is not used as compared with the example of FIG. 18, the prediction efficiency is lowered, but the method for setting the prediction vector candidate is simple, and thus there is a feature that the processing load is low. Further, the number M of prediction vector candidates may be encoded by a sequence level header or a picture level header. In this way, it is possible to set the number of candidates according to the temporal change in descriptor information, and it is possible to perform coding control in consideration of the code amount of prediction vector information and the prediction efficiency of motion vectors.

After step ST36, the inter prediction information in the encoding target pixel is output to the encoding unit 107, and the encoding unit 107 encodes the inter prediction information (step ST37). Here, the inter prediction information includes reference picture information that identifies a reference picture in which a prediction descriptor exists, the prediction vector information, and the difference vector.
Then, a difference descriptor signal is generated by subtracting the prediction descriptor signal generated by the prediction unit 103 from the descriptor signal of the encoding target pixel (step ST38), and a process of outputting this to the conversion unit 106 is performed. . The conversion unit 106 performs a process of converting the input differential descriptor signal (step ST39).

  On the other hand, if the referenced prediction mode is the direct encoding mode in step ST34, the descriptor signal of the encoding target pixel is output to the conversion unit 106 as it is, and the descriptor signal input by the conversion unit 106 is converted. Processing is performed (step ST40).

  Moreover, the conversion part 106 outputs the conversion signal produced | generated by step ST39 or step ST40 to the encoding part 107. Encoding section 107 encodes the input converted signal (step ST41).

  In the above description, the prediction mode and the conversion mode are described as being performed in units of blocks. However, the prediction mode and the conversion mode may be switched in units of pixels (feature points). In this case, step ST19 is not performed, and after step ST32, the prediction mode and the conversion mode of the pixel to be encoded are determined and encoded. By doing so, the code amount of the encoded data related to the prediction mode and the conversion mode increases, but more accurate prediction processing and conversion processing can be performed, and the code amount of the encoded data related to the conversion signal is reduced. be able to.

  In the above description, the prediction vector information is calculated and encoded in units of pixels (feature points), but may be calculated and encoded in units of code and other target blocks. In that case, prediction vector information shall be determined and encoded in step ST19. By doing so, although the prediction accuracy of the motion vector of each feature point decreases, the code amount of the encoded data related to the prediction vector information can be reduced.

Next, the operation of the image feature descriptor decoding apparatus will be described. FIG. 19 is a flowchart showing the processing contents (image feature descriptor decoding method) of the image feature descriptor decoding apparatus according to Embodiment 1 of the present invention. The image feature descriptor decoding apparatus performs the processing shown in FIG. 19 for each picture to realize decoding of descriptor information.
First, the decoding unit 201 decodes the header information of the picture data of the decoding target picture and the picture level header referred to by the header information of the picture data (step ST51). However, when the picture is the head of the sequence, the sequence level header is also decoded. Also, when the picture level header referred to in the decoding target picture has already been decoded, the picture level header is not decoded.

  Next, a reference picture list is generated based on the decoded header information (step ST52). Then, the decoded reference picture list is transferred to the descriptor information memory 204 as reference picture management information. Then, the descriptor presence flag which is one of the decoded header information is referred to (step ST53). When the descriptor presence flag is “1 (with descriptor)”, the block division of the picture is encoded with the descriptor information. The method is carried out by any one of the methods shown in FIGS. 7 to 11 described in the section 3 (step ST54). However, in the case of FIG. 11, the process up to the division of the reference block is performed. Then, the first reference block (decoding target block) to be decoded is set (step ST55), and decoding processing in units of reference blocks is performed in the processing order of FIGS. On the other hand, when the descriptor presence flag is “0 (descriptor formed)” in step ST53, the coding process of the descriptor information of the picture is finished, and the process proceeds to the next coded picture.

  As a decoding process for each decoding target block, the decoding unit 201 decodes the descriptor data of the decoding target block (step ST56). The descriptor data for each block to be decoded includes a descriptor presence flag for each block. When the descriptor presence flag is “1 (descriptor present)”, a prediction mode and a conversion mode further exist. When the descriptor presence / absence flag of the decoding target block is “1 (with descriptor)” (step ST57), the first processing target pixel in the decoding target block is set as the decoding target pixel (step ST58). Decoding of the descriptor information of the target pixel is performed (step ST59). Detailed processing for decoding the descriptor information of the decoding target pixel will be described later. Until the encoding process for the last pixel in the decoding target block is completed (step ST60), the descriptor information is encoded for each pixel in accordance with the same processing order as that of a predetermined image feature descriptor encoding apparatus such as a raster scan. Implement (step ST61).

  After decoding the descriptor information of all the pixels in the decoding target block, until the decoding process for the last block in the decoding target picture is completed (step ST62), the same as the predetermined image feature descriptor encoding device The descriptor information is decoded for each reference block in accordance with the processing order (step ST63). On the other hand, when the descriptor presence / absence flag of the block to be decoded is “0 (no descriptor)” in step ST57, the decoding process of the descriptor information in the block is terminated, and the process proceeds to the decoding process of the next processing block.

  However, in the case of the processing order of FIG. 11, the processing of step ST58 to step ST61 is performed only when the decoding target block is the minimum block. In other words, whether the reference block is hierarchically divided until the decoding target block becomes the minimum block, or no descriptor exists in the block, and one or more descriptors exist in the block in each block of each hierarchy The descriptor presence / absence flag indicating whether or not is decoded (step ST56).

  FIG. 20 is a flowchart showing the detailed processing content of step ST59. First, the descriptor data of the decoding target pixel is decoded (step ST71). This pixel unit descriptor data includes a pixel unit descriptor presence / absence flag. When the descriptor presence / absence flag indicates “1 (descriptor present)”, a conversion signal and inter prediction information further exist. When the descriptor presence / absence flag of the decoding target pixel indicates “1 (descriptor present)” (step ST72), the inverse transform unit 202 reverses the conversion signal based on the conversion mode of the decoding target block to which the decoding target pixel belongs. Conversion is performed to obtain a differential descriptor signal (step ST73).

When the prediction mode of the decoded pixel to be encoded is the inter prediction mode (step ST74), the prediction unit 203 performs inter prediction using the decoded inter prediction information (step ST75). Specifically, a prediction vector candidate is generated by the same method as the prediction vector candidate of the image feature descriptor encoding apparatus of the first embodiment described above, and the prediction vector is prediction vector information that is a part of the inter prediction information. A prediction vector is identified from the candidate index. Then, a difference vector that is also a part of the inter prediction information and the prediction vector are added to generate a motion vector. Then, for the reference picture indicated by the reference picture information that is also a part of the inter prediction information, the feature descriptor of the position (pixel) indicated by the motion vector is used as a prediction descriptor signal.
The adding unit 206 outputs the addition value of the difference descriptor signal output from the inverse transform unit 202 and the prediction descriptor signal output from the prediction unit 203 as a descriptor signal (step ST76).
On the other hand, if the prediction mode of the decoded pixel to be encoded is the direct encoding mode in step ST74, the difference descriptor signal output from the inverse transform unit 202 is output as a descriptor signal.

  Further, when the descriptor presence / absence flag of the pixel to be decoded is “0 (no descriptor)” in step ST72, the decoding process of the descriptor information at the pixel, that is, the process of step ST59 is ended.

  As is apparent from the above, according to the first embodiment, the prediction unit 103 of the image feature descriptor encoding apparatus performs motion compensation prediction using correlation in the time direction for each feature descriptor, The motion vector used for motion compensation prediction is also predicted from surrounding vectors to generate a difference vector, and the subtraction unit 105 generates a difference prediction descriptor signal using the prediction descriptor signal generated by the prediction unit 103. Since the conversion unit 106 converts the differential prediction descriptor signal into a conversion signal and the encoding unit 107 encodes the conversion signal and the motion difference vector, the motion vector information used for motion compensation is encoded with high efficiency. It is possible to predict the descriptor signal with high accuracy and encode the difference information with high efficiency, thereby realizing highly efficient encoding of descriptor information including the descriptor signal. There is that effect.

  Further, according to the first embodiment, the image feature descriptor decoding device capable of correctly decoding the encoded bitstream generated by the image feature descriptor encoding device and the image feature descriptor encoding method having the above-described effects. In addition, the image feature descriptor decoding method can be obtained.

Embodiment 2. FIG.
In Embodiment 1, prediction vector candidates for predictive encoding motion vectors generated by inter prediction are generated in units of encoding target blocks. However, in Embodiment 2, prediction vector candidates are generated in units of pixels. And FIG. 21 shows examples of candidate positions. A pixel adjacent to the encoding target pixel shown in FIG. 21 is set as a prediction vector candidate. However, a fixed vector is set for pixel positions that are not encoded.

In addition, among the prediction vector candidate pixels, a prediction vector candidate is set according to a predetermined procedure for pixels in which no motion vector exists. As an example of the predetermined procedure, among the encoded motion vectors, a method of setting a motion vector having a spatial distance closest to a pixel where no motion vector exists, or a fixed vector such as a zero vector is set. There is a technique. Examples of the spatial distance include Euclidean distance and Manhattan distance.
Thus, by making it possible to update prediction vector candidates in units of pixels, the accuracy of the prediction vector is improved, and high coding efficiency can be expected.

  21 (a) to 21 (f) differ in the number of prediction vector candidates. As the number of prediction vector candidates decreases, the motion vector prediction accuracy decreases. However, since the maximum value of the index decreases, the amount of encoded data related to the prediction vector information can be reduced.

  Furthermore, for the prediction vector candidate, when the outside of the encoding (decoding) target block is indicated, a fixed vector may be set so that the encoded motion vector is not used. In this case, it is possible to independently encode (decode) each block to be encoded (decoded). As a result, parallel processing can be realized and high-speed processing can be expected.

  In addition to the matters described above, the second embodiment implements the same configuration and processing as the first embodiment.

  In the present invention, within the scope of the invention, any combination of the embodiments, or any modification of any component in each embodiment, or omission of any component in each embodiment is possible. .

  The image feature descriptor encoding device, the image feature descriptor decoding device, the image feature descriptor encoding method, and the image feature descriptor decoding method according to the present invention can be applied to an image transmission system that realizes high encoding efficiency.

DESCRIPTION OF SYMBOLS 1 Input image pre-processing part, 2 Feature descriptor production | generation part, 3 Descriptor information encoding part, 4 Control part, 101 division | segmentation part, 102 Changeover switch, 103 Prediction part, 104 Descriptor information memory, 105 Subtraction part, 106 Conversion unit, 107 encoding unit, 108 prediction unit, 201 decoding unit, 202 inverse conversion unit, 203 prediction unit, 204 descriptor information memory, 205 selector switch, 206 addition unit, 207 prediction unit

Claims (6)

Block dividing means for dividing the input image into blocks;
In a block in which a descriptor presence / absence flag indicating that one or more descriptors are included in the block is true, when the prediction mode of the block is inter prediction, a motion vector is obtained for each feature point in the block. Prediction means that uses a coded descriptor signal on a reference picture indicated by
Conversion means for performing a conversion process on the difference value between the descriptor signal and the predicted descriptor signal according to the conversion mode of the block to generate a conversion signal;
Encode the descriptor presence flag, the prediction mode, the conversion mode, and the conversion signal of each feature point, and encode the descriptor presence flag, the prediction mode, the conversion mode, and the conversion signal of each feature point. Encoding means for generating a bitstream in which data is multiplexed;
With
The prediction means selects a prediction vector from one or more encoded motion vectors around the block that is a prediction vector candidate for each feature point, and a difference vector that is a difference between the motion vector and the prediction vector Produces
The encoding means encodes the prediction vector selection information and the difference vector for each feature point, and multiplexes the prediction vector selection information and the encoded data of the difference vector into a bitstream. An image feature descriptor encoding device.   2. The image feature descriptor encoding apparatus according to claim 1, wherein the prediction unit sets an encoded motion vector of a specific pixel among pixels adjacent to the block as the prediction vector candidate.   The predicting unit is an encoded motion vector that is spatially closest to the specific pixel when the encoded motion vector does not exist in the specific pixel among the pixels adjacent to the block as the prediction vector candidate. The image feature descriptor encoding apparatus according to claim 2, wherein: Decode the presence / absence flag, prediction mode, conversion mode, and conversion signal of each feature point related to each block divided from the encoded data multiplexed in the bitstream, and the prediction mode is inter prediction mode. If there is, decoding means for decoding prediction vector selection information and difference vectors for each feature point;
Differential signal generating means for generating a differential descriptor signal from the transformed signal related to the encoded block decoded by the decoding means;
When the prediction mode is inter prediction, for each feature point in the block, an encoded descriptor signal on the reference picture indicated by the motion vector calculated from the prediction vector selection information and the difference vector Predicting means with a predictive descriptor signal,
Image feature descriptor generation means for generating a descriptor signal by adding the difference descriptor signal generated by the difference signal generation means and the prediction descriptor signal generated by the prediction means;
With
The prediction means selects a prediction vector based on selection information of the decoded prediction vector from one or more encoded motion vectors around the block that is a prediction vector candidate for each feature point, and An image feature descriptor decoding apparatus characterized in that a motion vector is generated by adding the decoded difference vectors, and an encoded descriptor signal on a reference picture indicated by the motion vector is used as a prediction descriptor signal. A block division step for dividing the input image into blocks;
In a block in which a descriptor presence / absence flag indicating that one or more descriptors are included in the block is true, when the prediction mode of the block is inter prediction, a motion vector is obtained for each feature point in the block. A prediction step in which the encoded descriptor signal on the reference picture indicated by is a prediction descriptor signal;
A conversion step of performing a conversion process according to the conversion mode of the block on the difference value between the descriptor signal and the predicted descriptor signal to generate a conversion signal;
Encode the descriptor presence flag, the prediction mode, the conversion mode, and the conversion signal of each feature point, and encode the descriptor presence flag, the prediction mode, the conversion mode, and the conversion signal of each feature point. An encoding step for generating a bitstream in which data is multiplexed;
With
The prediction step selects, for each feature point, a prediction vector from one or more encoded motion vectors around the block that is a prediction vector candidate, and calculates a difference vector that is a difference between the motion vector and the prediction vector. Generate
The encoding step encodes the prediction vector selection information and the difference vector for each feature point, and multiplexes the prediction vector selection information and the encoded data of the difference vector into a bitstream. An image feature descriptor encoding method. Decode the presence / absence flag, prediction mode, conversion mode, and conversion signal of each feature point related to each block divided from the encoded data multiplexed in the bitstream, and the prediction mode is inter prediction mode. If there is, a decoding step for decoding prediction vector selection information and a difference vector of each feature point;
A differential signal generation step of generating a differential descriptor signal from the transformed signal related to the encoded block decoded by the decoding step;
When the prediction mode is inter prediction, for each feature point in the block, an encoded descriptor signal on the reference picture indicated by the motion vector calculated from the prediction vector selection information and the difference vector A prediction step with a prediction descriptor signal,
An image feature descriptor generation step of generating a descriptor signal by adding the difference descriptor signal generated by the difference signal generation step and the prediction descriptor signal generated by the prediction step;
With
The prediction step selects, for each feature point, a prediction vector based on selection information of the decoded prediction vector from one or more encoded motion vectors around the block that is a prediction vector candidate, and the prediction vector An image feature descriptor decoding method, wherein a motion vector is generated by adding the decoded difference vectors, and an encoded descriptor signal on a reference picture indicated by the motion vector is used as a prediction descriptor signal.

Images