WO2021152849A1 - データ処理装置及びデータ処理プログラム - Google Patents

データ処理装置及びデータ処理プログラム Download PDF

Info

Publication number
WO2021152849A1
WO2021152849A1 PCT/JP2020/003785 JP2020003785W WO2021152849A1 WO 2021152849 A1 WO2021152849 A1 WO 2021152849A1 JP 2020003785 W JP2020003785 W JP 2020003785W WO 2021152849 A1 WO2021152849 A1 WO 2021152849A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
data
unit
recognition
compression
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/003785
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
智規 久保田
鷹詔 中尾
康之 村田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujitsu Ltd
Original Assignee
Fujitsu Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujitsu Ltd filed Critical Fujitsu Ltd
Priority to PCT/JP2020/003785 priority Critical patent/WO2021152849A1/ja
Priority to JP2021574421A priority patent/JP7409400B2/ja
Publication of WO2021152849A1 publication Critical patent/WO2021152849A1/ja
Priority to US17/838,321 priority patent/US20220312019A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
    • H04N19/154—Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776—Validation; Performance evaluation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/98—Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
    • G06V10/993—Evaluation of the quality of the acquired pattern
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
    • H04N19/124—Quantisation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/176—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a block, e.g. a macroblock
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/189—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding
    • H04N19/192—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the adaptation method, adaptation tool or adaptation type used for the adaptive coding the adaptation method, adaptation tool or adaptation type being iterative or recursive
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression

Definitions

  • the present invention relates to a data processing apparatus and a data processing program.
  • the image data when recording or transmitting image data, the image data is compressed and the data size is reduced to reduce the recording cost and the transmission cost.
  • AI Artificial Intelligence
  • the conventional compression process is performed based on human visual characteristics, not based on AI motion analysis. For this reason, there are cases where the compression process is not performed at a sufficient compression level for the region that is not necessary for the recognition process by AI. Alternatively, the image quality of an important region in the recognition process by AI may be deteriorated, and sufficient recognition accuracy may not be obtained when decoding is performed.
  • the purpose is to realize compression processing suitable for recognition processing by AI.
  • the data processing device When the compression level is determined based on the degree of influence on the recognition result of each block when the recognition process is performed on the image data, the compression process is performed on the image data using the compression level. By doing so, when the encoding unit that generates the compressed data and the recognition result when the recognition processing is performed on the decoded data obtained by decoding the compressed data satisfy a predetermined condition, the block corresponding to the recognition target. Has a correction unit that corrects the compression level in the direction of increasing the compression level.
  • FIG. 1 is a first diagram showing an example of a system configuration of a compression processing system.
  • FIG. 2 is a diagram showing an example of a hardware configuration of an analysis device, an image compression device, or a data processing device.
  • FIG. 3 is a diagram showing an example of the functional configuration of the analyzer.
  • FIG. 4 is a diagram showing a specific example of the aggregation result.
  • FIG. 5 is a diagram showing a specific example of processing by the quantization value determination unit.
  • FIG. 6 is a diagram showing a specific example of processing by the foreground determination unit.
  • FIG. 7 is a diagram showing an example of the functional configuration of the image compression device.
  • FIG. 8 is a first diagram showing an example of the functional configuration of the data processing device.
  • FIG. 8 is a first diagram showing an example of the functional configuration of the data processing device.
  • FIG. 9 is a diagram showing a specific example of the processing of the quantization value correction unit.
  • FIG. 10 is a first flowchart showing an example of the flow of image compression processing by the compression processing system.
  • FIG. 11 is a second diagram showing an example of the system configuration of the compression processing system.
  • FIG. 12 is a second diagram showing an example of the functional configuration of the data processing device.
  • FIG. 13 is a first diagram showing a specific example of the processing of the analysis unit.
  • FIG. 14 is a second diagram showing a specific example of the processing of the analysis unit.
  • FIG. 15 is a second flowchart showing an example of the flow of image compression processing by the compression processing system.
  • FIG. 16 is a third diagram showing an example of the system configuration of the compression processing system.
  • FIG. 10 is a first flowchart showing an example of the flow of image compression processing by the compression processing system.
  • FIG. 11 is a second diagram showing an example of the system configuration of the compression processing system.
  • FIG. 12 is a second
  • FIG. 17 is a fourth diagram showing an example of the system configuration of the compression processing system.
  • FIG. 18 is a third diagram showing an example of the functional configuration of the data processing device.
  • FIG. 19 is a third flowchart showing an example of the flow of image compression processing by the compression processing system.
  • FIG. 20 is a fifth diagram showing an example of the system configuration of the compression processing system.
  • FIG. 21 is a sixth diagram showing an example of the system configuration of the compression processing system.
  • FIG. 22 is a fourth diagram showing an example of the functional configuration of the data processing device.
  • FIG. 23 is a fourth flowchart showing an example of the flow of image compression processing by the compression processing system.
  • FIG. 1 is a first diagram showing an example of a system configuration of a compression processing system.
  • the processing executed by the compression processing system is -The first phase of generating the determined quantization value map and -It can be roughly divided into a second phase in which the determined quantization value map is corrected, compression processing is performed using the corrected determined quantization value map, and the compressed data is stored.
  • 1a shows the system configuration of the compression processing system in the first phase
  • 1b shows the system configuration of the compression processing system in the second phase.
  • the compression processing system 100 in the first phase includes an image pickup device 110, an analysis device 120, and an image compression device 130.
  • the image pickup device 110 takes a picture at a predetermined frame cycle and transmits the image data to the analysis device 120.
  • the image data includes an object to be recognized.
  • the analysis device 120 has a trained model that performs recognition processing.
  • the analysis device 120 performs recognition processing by inputting image data into the trained model, and outputs the recognition result.
  • the analysis device 120 acquires each compressed data output by the image compression device 130 performing compression processing on the image data at different compression levels (quantization values), and decodes each compressed data. Generate each decrypted data. Further, the analysis device 120 performs recognition processing by inputting each decoded data into the trained model, and outputs the recognition result.
  • the analysis device 120 generates a map (referred to as an important feature map) showing the degree of influence on the recognition result by, for example, performing motion analysis of the trained model at the time of recognition processing by using the error back propagation method. .. Further, the analysis device 120 aggregates the degree of influence for each predetermined region (for each block used when the compression process is performed) based on the important feature map.
  • an important feature map a map showing the degree of influence on the recognition result by, for example, performing motion analysis of the trained model at the time of recognition processing by using the error back propagation method. ..
  • the analysis device 120 aggregates the degree of influence for each predetermined region (for each block used when the compression process is performed) based on the important feature map.
  • the quantization value map (variable) in which the quantization value is set in each block is sequentially transmitted to the image compression device 130 to perform compression processing at different compression levels (quantization value). , Instruct the image compression device 130.
  • the analysis device 120 generates an aggregated value graph for each block based on the aggregated value of the influence degree of each block aggregated each time the recognition process is performed on each decoded data.
  • the aggregated value graph is a graph showing the change of the aggregated value for each compression level (each quantized value). Further, the analysis device 120 determines the optimum compression level (quantization value) of each block based on each of the aggregated value graphs for each block.
  • the optimum quantization value of each block determined by the analyzer 120 will be referred to as a "determined quantization value”.
  • a map in which a determined quantization value is set in each block is referred to as a "determined quantization value map”.
  • the analysis device 120 transmits the determined quantization value map to the data processing device 140.
  • the analysis device 120 is suitable for the recognition process when performing the compression process on the image data.
  • the compression level can be determined.
  • the compression processing system 100 in the second phase includes an analysis device 120, an image compression device 130, a data processing device 140, and a storage device 150.
  • the analysis device 120 transmits the image data to the image compression device 130 and the data processing device 140.
  • the data processing device 140 performs compression processing on the image data transmitted from the analysis device 120 by using the determined quantization value map transmitted from the analysis device 120 in the first phase. Further, the data processing device 140 decodes the compressed data, performs recognition processing on the decoded data, and outputs the recognition result.
  • the data processing device 140 increases or decreases the quantization value of the block corresponding to the object to be recognized among the quantization values set in each block of the determined quantization value map in a predetermined step size. Recognize the decrypted data. Further, the data processing device 140 can compare the recognition result of the recognition result defined in advance based on the recognition result of the image data with the recognition result of each decoded data, and output the recognition result within the specified tolerance. Search for the maximum quantization value.
  • the data processing device 140 corrects the quantization value of the block corresponding to the object of the determination quantization value map using the searched maximum quantization value, and generates the corrected determination quantization value map. Further, the data processing device 140 transmits the generated corrected determined quantization value map to the image compression device 130.
  • the image compression device 130 performs compression processing on the image data using the transmitted corrected determined quantization value map, and stores the compressed data in the storage device 150.
  • the analysis device 120 when the analysis device 120 generates the determined quantization value map based on the degree of influence on the recognition result of each block, the recognition is performed based on the recognition result. Correct the quantization value of the block corresponding to the target object.
  • the compression level can be improved while maintaining the recognition result. That is, according to the data processing device 140 according to the first embodiment, it is possible to realize a compression process suitable for the recognition process by AI.
  • FIG. 2 is a diagram showing an example of the hardware configuration of the analysis device, the image compression device, or the data processing device.
  • the analysis device 120, the image compression device 130, or the data processing device 140 includes a processor 201, a memory 202, an auxiliary storage device 203, an I / F (Interface) device 204, a communication device 205, and a drive device 206.
  • the hardware of the analysis device 120, the image compression device 130, or the data processing device 140 is connected to each other via the bus 207.
  • the processor 201 has various arithmetic devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).
  • the processor 201 reads and executes various programs (for example, an analysis program, an image compression program, a data processing program, etc., which will be described later) on the memory 202.
  • the memory 202 has a main storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory).
  • the processor 201 and the memory 202 form a so-called computer, and the processor 201 realizes various functions by executing various programs read on the memory 202 (details of the various functions will be described later).
  • the auxiliary storage device 203 stores various programs and various data used when various programs are executed by the processor 201.
  • the I / F device 204 is a connection device that connects the operation device 210 and the display device 220, which are examples of external devices, to the analysis device 120, the image compression device 130, or the data processing device 140.
  • the I / F device 204 receives an operation on the analysis device 120, the image compression device 130, or the data processing device 140 via the operation device 210. Further, the I / F device 204 outputs the result of processing by the analysis device 120, the image compression device 130, or the data processing device 140, and displays the result via the display device 220.
  • the communication device 205 is a communication device for communicating with another device.
  • the analysis device 120 communicates with other devices such as the image pickup device 110, the image compression device 130, and the data processing device 140 via the communication device 205.
  • the image compression device 130 communicates with other devices such as the analysis device 120, the data processing device 140, and the storage device 150 via the communication device 205.
  • the data processing device 140 communicates with the analysis device 120 and the image compression device 130, which are other devices, via the communication device 205.
  • the drive device 206 is a device for setting the recording medium 230.
  • the recording medium 230 referred to here includes a medium such as a CD-ROM, a flexible disk, a magneto-optical disk, or the like that optically, electrically, or magnetically records information. Further, the recording medium 230 may include a semiconductor memory or the like for electrically recording information such as a ROM or a flash memory.
  • the various programs installed in the auxiliary storage device 203 are installed, for example, by setting the distributed recording medium 230 in the drive device 206 and reading the various programs recorded in the recording medium 230 by the drive device 206. Will be done.
  • the various programs installed in the auxiliary storage device 203 may be installed by being downloaded from the network via the communication device 205.
  • FIG. 3 is a diagram showing an example of the functional configuration of the analyzer.
  • the analysis device 120 has an analysis program installed, and when the program is executed, the analysis device 120 has an input unit 310, a CNN unit 320, a quantization value setting unit 330, and an output unit. Functions as 340. Further, the analysis device 120 functions as an important feature map generation unit 350, an aggregation unit 360, a quantization value determination unit 370, and a foreground determination unit 380.
  • the input unit 310 acquires the image data transmitted from the image pickup device 110 or the compressed data transmitted from the image compression device 130.
  • the input unit 310 notifies the CNN unit 320 and the output unit 340 of the acquired image data, decodes the acquired compressed data using a decoding unit (not shown), and notifies the CNN unit 320 of the decoded data.
  • the CNN unit 320 has a trained model, and by inputting image data or decoded data, performs recognition processing on the object to be recognized included in the image data or decoded data, and outputs the recognition result.
  • the recognition result includes a bounding box indicating the area of the recognized object, and the CNN unit 320 notifies the foreground determination unit 380 of the bounding box.
  • the quantization value setting unit 330 sets each compression level (each quantization value from the minimum quantization value (initial value) to the maximum quantization value) used when the image compression device 130 performs the compression process.
  • the quantization value map (variable) is sequentially notified to the output unit 340. Further, the quantization value setting unit 330 stores each set compression level (each quantization value) in the aggregation result storage unit 390.
  • the output unit 340 transmits the image data acquired by the input unit 310 to the image compression device 130. Further, the output unit 340 sequentially transmits each quantization value map (variable) notified from the quantization value setting unit 330 to the image compression device 130. Further, the output unit 340 transmits the determined quantization value map notified by the foreground determination unit 380 to the image compression device 130.
  • the important feature map generation unit 350 acquires the CNN part structure information when the trained model performs recognition processing on the image data or the decoded data, and uses the error backpropagation method based on the acquired CNN part structure information. By doing so, an important feature map is generated.
  • the important feature map generation unit 350 generates an important feature map by using, for example, a BP (Back Propagation) method, a GBP (Guided Back Propagation) method, or a selective BP method.
  • BP Back Propagation
  • GBP Guided Back Propagation
  • the error of each label is calculated from the classification probability obtained by performing the recognition process on the image data (or decoded data) whose recognition result is the correct label, and the error is back-propagated to the input layer.
  • This is a method of visualizing the featured part by imaging the magnitude of the gradient to be obtained.
  • the GBP method is a method of visualizing the feature portion by imaging only the positive value of the gradient information as the feature portion.
  • the selective BP method is a method in which only the error of the correct label is present, or only the error of the correct label is maximized, and then backpropagation is performed using the BP method or the GBP method. ..
  • the visualized feature part is a feature part that affects only the score information of the correct answer label.
  • the important feature map generation unit 350 uses the BP method, the GBP method, or the selective BP method in the CNN unit 320 from the input of the image data or the decoded data to the output of the recognition result. Analyze the signal flow and strength of each path. Thereby, according to the important feature map generation unit 350, it is possible to visualize which part of the input image data or the decoded data affects the recognition result to what extent (degree of influence).
  • the important feature map generation unit 350 analyzes the same information by analyzing the same information. Generate an important feature map.
  • the method of generating the important feature map by the backpropagation method is, for example, "Selvaraju, Ramprasaath R., et al.”
  • Grad-cam Visual explanations from deep networks via gradient-based localization.
  • ICCV Computer Vision
  • the aggregation unit 360 aggregates the degree of influence on the recognition result in block units based on the important feature map, and calculates the aggregated value of the degree of influence for each block. Further, the aggregation unit 360 associates the calculated aggregation value of each block with the quantized value and stores the aggregation result as the aggregation result in the aggregation result storage unit 390.
  • the quantization value determination unit 370 determines the optimum quantization value in each block based on the aggregation value graph of each block stored in the aggregation result storage unit 390. Further, the quantization value determination unit 370 notifies the foreground determination unit 380 of the quantization value map in which the determined optimum quantization value is set in each block.
  • the foreground determination unit 380 determines, among the blocks included in the bounding box notified by the CNN unit 320 and the blocks located on the outer periphery thereof, the blocks satisfying a predetermined condition are the foreground blocks. Further, the foreground determination unit 380 determines that a block other than the block determined to be the foreground block is a background block. Further, the foreground determination unit 380 maximizes the quantization value set in the block determined to be the background block among the quantization values set in each block.
  • the foreground determination unit 380 notifies the output unit 340 of a determined quantization value map including the quantization value set in the foreground block and the quantization value (maximized quantization value) set in the background block. do.
  • the method of determining the foreground block by the foreground determination unit 380 is not limited to this.
  • the foreground determination unit 380 may determine the foreground block based only on the aggregated value graph of each block, regardless of the bounding box notified by the CNN unit 320.
  • the foreground determination unit 380 may determine a block that satisfies a predetermined condition in the aggregated value graph as a foreground block, and may determine a block that does not satisfy the predetermined condition as a background block.
  • the foreground block may be determined using other information (eg, classification probability, etc.) independent of the bounding box.
  • the notification of the bounding box from the CNN unit 320 to the foreground determination unit 380 may be omitted.
  • FIG. 4 is a diagram showing a specific example of the aggregation result.
  • 4a shows an arrangement example of each block in the image data 410.
  • each block in the image data 410 has the same size and the same shape.
  • the block number of the upper left block of the image data is set to "block 1”
  • the block number of the lower right block is set to "block m”.
  • the aggregation result 420 includes "block number” and "quantized value” as information items.
  • the block number of each block in the image data 410 is stored in the "block number”.
  • the "quantization value” includes “no compression” indicating that the image compression device 130 does not perform compression processing, and the minimum quantization value ("Q") that the quantization value setting unit 330 sequentially sets in each block. Each quantization value from 1 ") to the maximum quantization value (“Q n ”) is stored.
  • the trained model performs the recognition process and performs the recognition process.
  • aggregated in the corresponding block The aggregated value is stored.
  • FIG. 5 is a diagram showing a specific example of processing by the quantization value determination unit.
  • the aggregated value graphs 510_1 to 510_m are generated by plotting the quantized value on the horizontal axis and the quantized value on the vertical axis and plotting the quantized value of each block included in the aggregated result 420. Will be done.
  • the aggregated value of each quantized value of each block used to generate the aggregated value graphs 510_1 to 510_m is, for example, -It may be adjusted using an offset value common to all blocks. ⁇ Absolute values may be taken and aggregated. -The aggregated values of other blocks may be processed based on the aggregated values of the blocks that are not attracting attention.
  • the change in the aggregated value when the minimum quantized value (Q 1 ) is changed to the maximum quantized value (Q n) is different for each block.
  • the quantization value determination unit 370 for example, ⁇ When the size of the aggregated value exceeds a predetermined threshold, or ⁇ When the amount of change in the aggregated value exceeds a predetermined threshold, or ⁇ When the slope of the aggregated value exceeds a predetermined threshold, or ⁇ When the change in the slope of the aggregated value exceeds a predetermined threshold When any of the above conditions is satisfied, the optimum quantization value of each block is determined, and a quantization value map is generated.
  • the quantization value map 530 shows how B 1 Q to B m Q are determined as the optimum quantization values of blocks 1 to m and are set in the corresponding blocks, respectively.
  • the quantization value determination unit 370 determines the quantization value as follows, for example. -When the size of the block used for compression processing is larger than the size of the block used for aggregation processing The average value of the quantization values (or the average value of the quantization values based on the aggregated value of each block during aggregation processing) included in the blocks used for compression processing. , Minimum value, maximum value, and values processed by other indexes) are used as the quantization value of each block used in the compression process.
  • the aggregation of blocks at the time of aggregation Use value-based quantization values.
  • the process of actually calculating the aggregated value may be performed based on only one quantization value (one compression level). In that case, by assuming different quantization values (different compression levels) and measuring the difference or change between the aggregated value corresponding to the assumed quantization value and the aggregated value corresponding to the actual quantization value, the aggregation is performed. The value shall be calculated.
  • the image quality of the decoded data for the assumed quantization value may be better or worse than the image quality of the decoded data for the actual quantization value (compression level).
  • the assumed quantization value is a quantization value that makes it easy to estimate the state of the aggregated value. For example, when comparing the aggregated value corresponding to the actual quantization value with the image data that has not been compressed, the aggregated value of the image data that has not been compressed is generally the actual quantum. It is smaller than the aggregated value corresponding to the quantized value.
  • the aggregated value corresponding to the actual quantization value may be calculated using the decrypted data obtained by decoding the compressed data that has been compressed using the actual quantization value.
  • it may be calculated using image data that has been subjected to image processing (for example, low-pass filter processing) that produces the same effect.
  • the aggregated value corresponding to the actual quantization value can be calculated using image data that has been operated beyond the range of image quality change that can be controlled within the range of the maximum and minimum values of the quantization value. good. For example, it may be calculated using image data that has undergone image processing that exceeds the maximum value of the quantization value that can be specified in the moving image coding process.
  • the threshold value applied when evaluating the aggregated value graph may be different or the same for each block. Further, the threshold value applied when evaluating the aggregated value graph may or may not be adjusted based on the score information of the recognition result, for example.
  • the threshold value applied when evaluating the aggregated value graph may be automatically determined. Specifically, based on the information that can be acquired during the recognition process, the information that can be acquired from the image data, the value obtained by statistically processing them, the amount of compressed data and its transition, or other processing. It may be determined automatically based on the information that can be acquired.
  • FIG. 6 is a diagram showing a specific example of processing by the foreground determination unit.
  • the foreground determination unit 380 is notified by the quantization value determination unit 370 of the quantization value map 530 in which the quantization value is set in each block. Further, the foreground determination unit 380 is notified by the CNN unit 320 of the bounding boxes (bounding boxes 611 and 612 in the example of FIG. 6) indicating the area of the object.
  • the foreground determination unit 380 determines, for example, that the block included in the bounding box 611 is a foreground block. Further, the foreground determination unit 380 determines whether or not the outer peripheral block of the bounding box 611 is a foreground block based on the aggregated value graph.
  • the foreground determination unit 380 determines, for example, that the block included in the bounding box 612 is a foreground block. Further, the foreground determination unit 380 determines whether or not the outer peripheral block of the bounding box 612 is a foreground block based on the aggregated value graph.
  • the method of determining whether or not it is a foreground block by the foreground determination unit 380 is not limited to this, and for example, it may be determined whether or not it is a foreground block based only on the aggregated value graph. Alternatively, it may be determined whether or not the block is a foreground block based on the classification probability of each block included in the recognition result notified by the CNN unit 320.
  • the foreground determination unit 380 does not modify the quantization value set in the block determined to be the foreground block.
  • the foreground determination unit 380 determines that a block other than the foreground block is a background block.
  • the foreground determination unit 380 generates a determined quantization value map by maximizing the quantization value set in the block determined to be the background block.
  • the determined quantized value map 620 shows an example of the determined quantized value map generated by the foreground determination unit 380.
  • the white block included in the determined quantization value map 620 is a block determined to be a foreground block by the foreground determination unit 380, and the quantization value determined by the quantization value determination unit 370 is set.
  • the shaded block included in the determined quantization value map 620 is a block determined to be a background block by the foreground determination unit 380, and the maximized quantization value is set.
  • FIG. 7 is a first diagram showing an example of the functional configuration of the image compression device.
  • the image compression device 130 has an image compression program installed, and when the program is executed, the image compression device 130 functions as the coding unit 720.
  • the coding unit 720 includes a difference unit 721, an orthogonal conversion unit 722, a quantization unit 723, an entropy coding unit 724, an inverse quantization unit 725, and an inverse orthogonal conversion unit 726. Further, the coding unit 720 includes an addition unit 727, a buffer unit 728, an in-loop filter unit 729, a frame buffer unit 730, an in-screen prediction unit 731, and an inter-screen prediction unit 732.
  • the difference unit 721 calculates the difference between the image data (for example, image data 410) and the predicted image data, and outputs the predicted residual signal.
  • the orthogonal transform unit 722 executes the orthogonal transform process on the predicted residual signal output by the difference unit 721.
  • the quantization unit 723 quantizes the predicted residual signal that has undergone orthogonal transformation processing, and generates a quantization signal.
  • the quantization unit 723 generates a quantization signal using a quantization value map (variable) sequentially transmitted from the analyzer 120 in the first phase, and a data processing device in the second phase.
  • a quantization signal is generated using the corrected determined quantization value map transmitted from 140.
  • the entropy coding unit 724 generates compressed data by performing entropy coding processing on the quantized signal.
  • the dequantization unit 725 dequantizes the quantization signal.
  • the inverse orthogonal transform unit 726 executes an inverse orthogonal transform process on the inverse quantized quantized signal.
  • the addition unit 727 generates reference image data by adding the signal output from the inverse orthogonal transform unit 726 and the predicted image data.
  • the buffer unit 728 stores the reference image data generated by the addition unit 727.
  • the in-loop filter unit 729 filters the reference image data stored in the buffer unit 728.
  • DB Deblocking filter
  • SAO sample Adaptive Offset filter
  • ALF Adaptive loop filter
  • the frame buffer unit 730 stores the reference image data filtered by the in-loop filter unit 729 in frame units.
  • the in-screen prediction unit 731 makes an in-screen prediction based on the reference image data and generates the predicted image data.
  • the inter-screen prediction unit 732 performs motion compensation between frames using the input image data (for example, image data 410) and reference image data, and generates predicted image data.
  • the predicted image data generated by the in-screen prediction unit 731 or the inter-screen prediction unit 732 is output to the difference unit 721 and the addition unit 727.
  • the coding unit 720 refers to MPEG-2, MPEG-4, H.M.
  • the compression process is performed using an existing moving image coding method such as 264 or HEVC.
  • the compression process by the coding unit 720 is not limited to these moving image coding methods, and may be performed by using an arbitrary coding method in which the compression rate is controlled by parameters such as a quantization value.
  • FIG. 8 is a first diagram showing an example of the functional configuration of the data processing device.
  • a data processing program is installed in the data processing device 140, and when the program is executed, the data processing device 140 has a coding unit 810, a decoding unit 820, a CNN unit 830, and a quantum. It functions as a conversion value correction unit 840.
  • the coding unit 810 performs compression processing on the image data transmitted from the analysis device 120 using the determined quantization value map transmitted from the analysis device 120, and generates compressed data. Further, when the coding unit 810 is notified by the quantization value correction unit 840 of an instruction for increasing or decreasing the quantization value of the foreground block of the determined quantization value map, the coding unit 810 notifies the image data of the quantization value. Compressed data is generated by performing compression processing using the determined quantization value map in which is increased or decreased.
  • the coding unit 810 notifies the decoding unit 820 of the generated compressed data each time the compressed data is generated, based on the instruction from the quantization value correction unit 840.
  • the function of the coding unit 810 is basically the same as the function of the coding unit 720 of the image compression device 130, detailed description thereof will be omitted here.
  • the decoding unit 820 decodes each compressed data and generates the decrypted data. Further, the decoding unit 820 notifies the CNN unit 830 of the decoded data.
  • the CNN unit 830 has a trained model, and by inputting the decoded data, the object to be recognized included in the decoded data is recognized and the recognition result is output. Further, the CNN unit 830 notifies the quantization value correction unit 840 of the score information included in the output recognition result.
  • the CNN unit 830 performs recognition processing each time the decoding unit 820 notifies the decoding data, and notifies the quantized value correction unit 840 of the score information.
  • the CNN section 830 -When the coding unit 810 generates compressed data by performing compression processing using the determined quantized value map, and When the decoding unit 820 inputs the decrypted data generated by decoding the compressed data to the CNN unit 830, The score information included in the recognition result output by performing the recognition process is notified to the quantization value correction unit 840 as "reference score information".
  • the coding unit 810 When the coding unit 810 generates compressed data by performing compression processing using the determined quantization value map in which the quantization value of the foreground block is increased or decreased, and When the decoding unit 820 inputs the decrypted data generated by decoding the compressed data to the CNN unit 830, The score information included in the recognition result output by performing the recognition process is notified to the quantization value correction unit 840 as "score information".
  • the quantization value correction unit 840 is an example of the correction unit, and among the quantization values set in each block of the determined quantization value map notified by the analyzer 120, the quantization value set in the foreground block is set. Increase or decrease in a predetermined step size.
  • the quantization value correction unit 840 is set in the foreground block when the reference score information notified from the CNN unit 830 is equal to or higher than a predetermined threshold value (when the predetermined first condition is satisfied). The process of increasing the quantization value by a predetermined step size is started.
  • the quantization value correction unit 840 determines that the score information notified from the CNN unit 830 is within the permissible range specified for the reference score information (predetermined). (As long as the second condition of) is satisfied, the process of increasing the quantization value is continued.
  • the quantization value correction unit 840 performs a process of increasing the quantization value while the score information notified from the CNN unit 830 is equal to or higher than a predetermined threshold value (as long as the predetermined first condition is satisfied). continue.
  • the quantization value correction unit 840 is set in the foreground block. The process of reducing the quantized value by a predetermined step size is started.
  • the quantization value correction unit 840 does not satisfy the predetermined first condition (the predetermined first condition is not satisfied) while the score information notified from the CNN unit 830 is less than a predetermined threshold value. In the meantime), the process of reducing the quantization value is continued.
  • the quantization value correction unit 840 corrects the quantization value of the foreground block to the quantization value at the time of completion, and corrects it.
  • the post-determined quantization value map is transmitted to the image compression device 130.
  • the step size when the quantization value correction unit 840 increases or decreases the quantization value is “1" (or “-1”).
  • the step size when increasing or decreasing the quantization value by the quantization value correction unit 840 is "1" or more (or “-1” or less) even if it is “1” (or “-1”). You may.
  • the permissible range defined based on the reference score information is compared with the score information. It was explained as.
  • the method for determining whether or not to continue the process of increasing the quantization value is not limited to this.
  • the IoU Intersection over Union
  • the IoU calculated based on the bounding box included in the recognition result output from the CNN unit 830 may be compared with the predetermined allowable range of the IoU.
  • the quantization value correction unit 840 may be able to control how strictly the process of increasing the quantization value is performed according to the application application, the required recognition accuracy, and the like.
  • FIG. 9 is a diagram showing a specific example of processing by the data processing device.
  • the horizontal axis 900 indicates the quantized value.
  • reference numeral 901 indicates a quantization value set in block a_1 in the determined quantization value map among the 24 blocks (blocks a_1 to block a_24) included in the foreground block.
  • reference numeral 902 indicates a quantization value set in block a_24 in the determined quantization value map among the 24 blocks (blocks a_1 to block a_24) included in the foreground block.
  • the quantization value set in the block a_1 is "33", and the quantization value set in the block a_24 is "32".
  • the reference score information when the compression process is performed using these quantized values and the decrypted data obtained by decoding the compressed data is subjected to the recognition process is predetermined. It indicates that it is determined that the value is equal to or higher than the threshold value (the predetermined first condition is satisfied).
  • the score information is the predetermined first. Alternatively, it indicates that it is determined that the second condition is not satisfied (see the right end of reference numeral 903).
  • the quantization value correction unit 840 changes the quantization value of the block a_1 from “33” to "41” and the quantum of the block a_24 as shown in the corrected determined quantization value map 920.
  • the quantization value is corrected from "32" to "41".
  • reference numeral 911 indicates a quantization value set in block b_1 in the determined quantization value map among the 24 blocks (blocks b_1 to block b_24) included in the foreground block.
  • reference numeral 912 indicates a quantization value set in block b_24 in the determined quantization value map among the 24 blocks (blocks b_1 to block b_24) included in the foreground block.
  • the quantization value set in the block b_1 is "28", and the quantization value set in the block b_24 is "29".
  • the reference score information when the compression process is performed using these quantized values and the decrypted data obtained by decoding the compressed data is subjected to the recognition process is predetermined. It is less than the threshold value (indicating that it is determined that the predetermined first condition is not satisfied.
  • the score information is the predetermined first. It is shown that it is determined that the condition of the above condition is satisfied (see the left end of reference numeral 913).
  • the quantization value correction unit 840 changes the quantization value of the block b_1 from “28” to “20” and the quantum of the block b_24 as shown in the corrected determined quantization value map 920.
  • the quantized value is corrected from "29" to "20".
  • the method of increasing the quantization value of each block is not limited to this.
  • a process of specifying the minimum quantization value among the quantization values of each block and increasing only the block of the specified minimum quantization value may be sequentially performed.
  • the quantization value of block a_10 is "30"
  • the quantization value of block a_11 is "32”
  • the quantization value of block a_12 is "36".
  • it is increased as (31, 33, 37), (32, 34, 38), ..., But according to the above increasing method, (31, 32, 36). ), (32, 32, 36), (33, 33, 36), ...
  • the reference score information may be specified for each object, and the quantization value may be corrected based on the recognition result of each object.
  • the quantization value of each block is uniformly increased and the recognition processing is performed on the object A and the object B
  • the quantization value of the block included in the object A is "40”
  • the object A can be recognized, but when the quantization value is "41” or more, the object A cannot be recognized.
  • the quantization value of the block included in the object B is "30”
  • the object B can be recognized, but when the quantization value is "31” or more, the object B cannot be recognized.
  • the quantization value of the block included in the object A is corrected to "40"
  • the quantization value of the block included in the object B is corrected to "30”.
  • the quantization value is corrected individually for each object, the consistency of the entire image data will not match, and there is a possibility that unrecognizable objects will occur. In such a case, it may be corrected by using the maximum value of the logical product condition of the quantization value that can recognize all the objects.
  • the quantization value of the block included in the object B is fixed at the quantization value at the time when the search end condition is satisfied, and the quantization value of the block included in the object A is fixed until the search end condition is satisfied.
  • the quantization value may be continuously increased.
  • FIG. 10 is a first flowchart showing an example of the flow of image compression processing by the compression processing system.
  • step S1001 the input unit 310 of the analysis device 120 acquires image data, and in step S1002, the CNN unit 320 of the analysis device 120 performs recognition processing on the acquired image data and outputs the recognition result.
  • step S1003 the quantization value setting unit 330 of the analyzer 120 sequentially sets each quantization value from the minimum quantization value (Q 1 ) to the maximum quantization value (Q n ), and the output unit 340 sets each quantization value in sequence.
  • Each quantization value map (variable) is transmitted to the image compression device 130. Further, the image compression device 130 performs compression processing on the image data using each transmitted quantization value map (variable), and generates each compression data.
  • step S1004 the input unit 310 of the analysis device 120 decodes each compressed data generated by the image compression device 130.
  • the CNN unit 320 of the analysis device 120 performs recognition processing on each decoded data.
  • the important feature map generation unit 350 of the analysis device 120 generates each important feature map showing the degree of influence on the recognition result of each region of the decoded data based on the CNN part structure information.
  • step S1005 the aggregation unit 360 of the analysis device 120 aggregates the influence degree of each region for each important feature map in block units. Further, the aggregation unit 360 of the analysis device 120 stores the aggregation result in the aggregation result storage unit 390 in association with each compression level (quantized value).
  • step S1006 the quantization value determination unit 370 of the analyzer 120 determines the quantization value in block units based on the aggregated value graph of each block, and generates a quantization value map.
  • step S1007 the foreground determination unit 380 of the analyzer 120 maximizes the quantization value set in the background block among the generated quantization value maps, and generates a determined quantization value map.
  • step S1008 the data processing device 140 performs recognition processing while increasing or decreasing the quantization value set in the foreground block among the quantization values set in each block of the determined quantization value map.
  • step S1009 the data processing device 140 corrects the quantization value set in the foreground block of the determination quantization value map based on the recognition result, and generates the corrected determination quantization value map.
  • step S1010 the image compression device 130 performs compression processing on the image data using the corrected determined quantization value map, and stores the compressed data in the storage device 150.
  • the data processing apparatus is determined and quantized based on the degree of influence on the recognition result of each block when the recognition processing is performed on the image data.
  • compression processing is performed using the determined quantization value map.
  • the data processing apparatus is a foreground block corresponding to the recognition target when the recognition result when the recognition processing is performed on the decrypted data obtained by decoding the compressed data satisfies a predetermined condition. Is corrected in the direction of increasing the compression level (quantization value).
  • the data processing apparatus corrects the quantization value determined based on the degree of influence on the recognition result in the direction of increasing based on the recognition result.
  • the compression level can be improved while maintaining the recognition accuracy. That is, according to the first embodiment, it is possible to realize a compression process suitable for the recognition process by AI.
  • the recognition accuracy of such image data is improved by first changing the image data itself. Subsequently, the quantized value of the changed image data is determined based on the degree of influence on the recognition result, and compression processing is performed using the determined quantized value.
  • the second embodiment it is possible to improve the compression level of the image data while improving the recognition accuracy.
  • the second embodiment will be described focusing on the differences from the first embodiment.
  • FIG. 11 is a second diagram showing an example of the system configuration of the compression processing system.
  • the processing executed by the compression processing system 1100 is ⁇
  • the first phase of changing image data and -A determined quantization value map is generated based on the changed image data, and compression processing is performed using the generated determined quantization value map, which is roughly divided into the second phase in which the compressed data is stored. Can be done.
  • 11a shows the system configuration of the compression processing system 1100 in the first phase
  • 11b shows the system configuration of the compression processing system 1100 in the second phase.
  • the compression processing system 1100 in the first phase includes an imaging device 110 and a data processing device 1110.
  • the processing by the imaging device 110 is the same as the processing by the imaging device 110 described with reference to 1a of FIG. 1 in the first embodiment, and thus the description thereof will be omitted here.
  • the data processing device 1110 performs recognition processing on the image data transmitted from the image pickup device 110. Further, the data processing device 1110 determines whether or not the score information included in the recognition result satisfies a predetermined condition, and if it is determined that the score information does not satisfy the condition, the image data is changed so as to maximize the score information. Then, the changed image data is transmitted to the analysis device 120.
  • the data processing device 1110 determines that the score information included in the recognition result satisfies a predetermined condition, the data processing device 1110 transmits the image data to the analysis device 120 without changing the image data.
  • the compression processing system 1100 in the second phase includes an analysis device 120, an image compression device 130, and a storage device 150.
  • the analysis device 120 has a trained model that performs recognition processing.
  • the analysis device 120 performs the recognition process by inputting the image data or the changed image data into the trained model, and outputs the recognition result. Further, the analysis device 120 acquires each compressed data output by the image compression device 130 by performing compression processing on the image data or the changed image data at different compression levels (quantization values), and each compressed data. Each decrypted data is generated by decoding. Further, the analysis device 120 performs recognition processing by inputting each decoded data into the trained model, and outputs the recognition result.
  • the analysis device 120 generates an important feature map by performing motion analysis of the trained model at the time of recognition processing by using, for example, the error back propagation method. Further, the analysis device 120 aggregates the degree of influence for each block based on the important feature map.
  • the quantization value map (variable) in which the quantization value is set in each block is sequentially transmitted to the image compression device 130 to perform compression processing at different compression levels (quantization value). , Instruct the image compression device 130.
  • the analysis device 120 generates an aggregated value graph for each block based on the aggregated value of the influence degree of each block calculated every time the recognition process is performed on each decoded data. Further, the analysis device 120 determines the optimum compression level (quantization value) of each block based on each of the aggregated value graphs for each block, and generates a determined quantization value map.
  • the image compression device 130 performs compression processing on the image data or the changed image data using the generated determined quantization value map, and stores the compressed data in the storage device 150.
  • FIG. 12 is a second diagram showing an example of the functional configuration of the data processing device. Similar to the first embodiment, a data processing program is installed in the data processing device 1110, and when the program is executed, the data processing device 1110 functions as a CNN unit 1210 and a determination unit 1220. Further, the data processing device 1110 functions as an analysis unit 1230 and an image data change unit 1240.
  • the CNN unit 1210 has a trained model, and by inputting image data, it performs recognition processing on the object to be recognized included in the image data and outputs the recognition result.
  • the determination unit 1220 determines whether or not the score information (an example of information related to the recognition accuracy of image data) included in the recognition result output from the CNN unit 1210 satisfies a predetermined condition (for example, a predetermined threshold value or more). (Determine whether or not it is). When it is determined that the score information included in the recognition result satisfies a predetermined condition, the determination unit 1220 notifies the image data change unit 1240 of the determination result. On the other hand, when it is determined that the score information included in the recognition result does not satisfy the predetermined condition, the determination unit 1220 notifies the analysis unit 1230 of the determination result.
  • a predetermined condition for example, a predetermined threshold value or more
  • the analysis unit 1230 acquires the image data and analyzes the acquired image data. Further, the analysis unit 1230 notifies the image data change unit 1240 of the change information for maximizing the score information generated by analyzing the image data. Alternatively, the analysis unit 1230 notifies the image data change unit 1240 of the image data (changed image data) for maximizing the score information generated by analyzing the image data.
  • the image data changing unit 1240 is an example of the changing unit.
  • the image data change unit 1240 transmits the image data to the analysis device 120 without changing the image data.
  • the image data change unit 1240 changes the image data based on the notified change information and transmits the changed image data to the analysis device 120.
  • the image data changing unit 1240 transmits the changed image data to the analysis device 120.
  • FIG. 13 is a first diagram showing a specific example of processing by the analysis unit.
  • the analysis unit 1230 includes, for example, a refined image generation unit 1310, an important feature index map generation unit 1320, a specific unit 1340, and a detailed analysis unit 1350.
  • the refined image generation unit 1310 has an image refiner unit 1311, an image error calculation unit 1312, an inference unit 1313, and a score error calculation unit 1314.
  • the image refiner unit 1311 generates refined image data from the image data by learning using CNN as an image data generation model, for example.
  • the image refiner unit 1311 changes the image data so that the score information of the correct label is maximized when the inference unit 1313 performs the recognition process using the generated refined image data. Further, the image refiner unit 1311 generates refined image data so that the amount of change from the image data (difference between the refined image data and the image data) is small, for example. As a result, according to the image refiner unit 1311, it is possible to obtain refined image data that is visually close to the image data before the change.
  • -The error (score error) between the score information when recognition processing is performed using the generated refined image data and the score information that maximizes the score information of the correct label.
  • image difference value which is the difference between the generated refined image data and the image data, CNN learning is performed so as to minimize.
  • the image error calculation unit 1312 calculates the difference between the image data and the refined image data output from the image refiner unit 1311 during CNN learning, and inputs the image difference value to the image refiner unit 1311.
  • the image error calculation unit 1312 calculates an image difference value by, for example, performing a pixel-by-pixel difference (L1 difference) or a SSIM (Structural Similarity) calculation, and inputs the image difference value to the image refiner unit 1311.
  • the reasoning unit 1313 has a learned CNN that performs recognition processing by inputting refined image data generated by the image refiner unit 1311 and outputs score information.
  • the score information output by the inference unit 1313 is notified to the score error calculation unit 1314.
  • the score error calculation unit 1314 calculates the error between the score information notified by the inference unit 1313 and the score information that maximizes the score information of the correct answer label, and notifies the image refiner unit 1311 of the score error.
  • the score error notified by the score error calculation unit 1314 is used for learning CNN in the image refiner unit 1311.
  • the refined image output from the image refiner unit 1311 during learning of the CNN of the image refiner unit 1311 is stored in the refined image storage unit 1315.
  • the refined image data performed and when the score information of the correct answer label output from the inference unit 1313 is maximized is hereinafter referred to as "score maximized refined image data".
  • the important feature index map generation unit 1320 has an important feature map generation unit 1321, a deterioration scale map generation unit 1322, and a superimposition unit 1323.
  • the important feature map generation unit 1321 acquires the inference unit structure information when the inference unit 1313 performs the recognition process by inputting the score maximization refined image data from the inference unit 1313. In addition, the important feature map generation unit 1321 generates an important feature map based on the inference unit structural information by using the BP method, the GBP method, or the selective BP method.
  • the deterioration scale map generation unit 1322 generates a "deterioration scale map" based on the image data and the score maximization refined image data.
  • the deterioration scale map is a map showing the changed portion and the degree of change of each changed portion when the image data is changed to the score maximizing refined image data.
  • the superimposition unit 1323 generates the important feature index map 1330 by superimposing the important feature map generated by the important feature map generation unit 1321 and the deterioration scale map generated by the deterioration scale map generation unit 1322.
  • the important feature index map 1330 is a map that visualizes the degree of influence of the image data on the recognition result of each region.
  • the specific unit 1340 divides the image data in units of super pixels, for example, and aggregates the important feature index map 1330 in units of super pixels.
  • the identification unit 1340 identifies the super pixel whose image data is to be changed based on the aggregation result. Further, the specific unit 1340 notifies the detailed analysis unit 1350 of the important feature index map 1330 included in the specified super pixel among the important feature index maps 1330 as a cause region of erroneous recognition.
  • the detailed analysis unit 1350 generates change information for changing the image data on a pixel-by-pixel basis based on the cause region generated by the specific unit 1340, and notifies the image data change unit 1240.
  • the image data changing unit 1240 changes the image data on a pixel-by-pixel basis based on the changed information, and transmits the changed image data to the analysis device 120.
  • FIG. 14 is a second diagram showing a specific example of processing by the analysis unit.
  • the analysis unit 1230 has, for example, a refined image generation unit 1310.
  • the refined image generation unit 1310 has an image refiner unit 1311, an image error calculation unit 1312, an inference unit 1313, and a score error calculation unit 1314.
  • the function of each part of the refined image generation unit 1310 is the same as the function of each part of the refined image generation unit 1310 shown in FIG. However, in the case of FIG. 14, the score maximizing refined image stored in the refined image storage unit 1315 is read out by the image data changing unit 1240 as the changed image data.
  • the image data changing unit 1240 transmits the score maximizing refined image read from the refined image storage unit 1315 to the analysis device 120 as the changed image data.
  • FIG. 15 is a second flowchart showing an example of the flow of image compression processing by the compression processing system.
  • step S1501 the CNN unit 1210 of the data processing device 1110 acquires image data from the image pickup device 110.
  • step S1502 the CNN unit 1210 of the data processing device 1110 performs recognition processing on the acquired image data and outputs the recognition result.
  • step S1503 the determination unit 1220 of the data processing device 1110 determines whether or not the image data needs to be changed by determining whether or not the score information included in the recognition result satisfies a predetermined condition. If it is determined in step S1503 that the predetermined condition is not satisfied (yes in step S1503), it is determined that the image data needs to be changed, and the process proceeds to step S1504.
  • step S1504 the analysis unit 1230 of the data processing device 1110 generates change information for changing the image data so that the score information is maximized. Further, the image data changing unit 1240 of the data processing device 1110 changes the image data based on the generated change information, and transmits the changed image data to the analysis device 120.
  • the analysis unit 1230 of the data processing device 1110 generates a score maximizing refined image by changing the image data so as to maximize the score information, and notifies the image data changing unit 1240. Further, the image data changing unit 1240 of the data processing device 1110 transmits the score maximizing refined image to the analysis device 120 as the changed image data.
  • step S1503 when it is determined in step S1503 that the predetermined condition is satisfied (No in step S1503), it is determined that the image data does not need to be changed, and the analysis device 120 is used without changing the image data. Send.
  • step S1505 the CNN unit 320 of the analysis device 120 performs recognition processing on the changed image data (or image data) transmitted from the image data changing unit 1240, and outputs the recognition result.
  • step S1506 the quantization value setting unit 330 of the analyzer 120 sequentially sets each quantization value from the minimum quantization value (Q 1 ) to the maximum quantization value (Q n ), and the output unit 340 sets each quantization value in sequence.
  • Each quantization value map (variable) is transmitted to the image compression device 130. Further, the image compression device 130 performs compression processing on the image data using each transmitted quantization value map (variable), and generates each compression data.
  • step S1507 the input unit 310 of the analysis device 120 decodes each compressed data generated by the image compression device 130.
  • the CNN unit 320 of the analysis device 120 performs recognition processing on each decoded data.
  • the important feature map generation unit 350 of the analysis device 120 generates each important feature map showing the degree of influence on the recognition result of each region of the decoded data based on the CNN part structure information.
  • step S1508 the aggregation unit 360 of the analysis device 120 aggregates the influence degree of each region for each important feature map in block units. Further, the aggregation unit 360 of the analysis device 120 stores the aggregation result in the aggregation result storage unit 390 in association with each compression level (each quantization value).
  • step S1509 the quantization value determination unit 370 of the analyzer 120 determines the quantization value in block units based on the aggregated value graph of each block, and generates a quantization value map.
  • step S1510 the foreground determination unit 380 of the analyzer 120 maximizes the quantization value set in the background block among the generated quantization value maps, and generates a determined quantization value map.
  • step S1511 the image compression device 130 performs compression processing on the changed image data (or image data) using the determined quantization value map, and stores the compressed data in the storage device 150.
  • the data processing device performs recognition processing on the image data acquired from the image pickup device 110, and determines whether or not the score information satisfies a predetermined condition. do. Further, the data processing apparatus according to the second embodiment changes the image data so that the score information is maximized when it is determined that the predetermined condition is not satisfied.
  • the recognition accuracy can be improved even when the image data having low recognition accuracy is acquired.
  • the determined quantization value map is generated based on the changed image data, according to the second embodiment, the determined quantization value map in which a high quantization value is set can be generated.
  • the compression level can be improved while improving the recognition accuracy. That is, according to the data processing apparatus according to the second embodiment, it is possible to realize a compression process suitable for the recognition process by AI.
  • the third embodiment it is determined whether or not the image data needs to be changed in the process of increasing the quantization value when generating the determined quantization value map, and the image data When it is determined that the change is necessary, the image data is changed.
  • the third embodiment it is possible to improve the compression level while improving the recognition accuracy as in the second embodiment.
  • the third embodiment will be described focusing on the differences from the second embodiment.
  • the processing executed by the compression processing system is -In order to generate a determined quantization value map, the first phase of performing compression processing at different compression levels (quantization values) and monitoring the aggregated value graph, and -A second phase in which the image data is changed and the same processing is performed on the changed image data when it is determined that the image data needs to be changed based on the aggregated value graph.
  • 16a shows the system configuration of the compression processing system 1600 in the first phase
  • 16b shows the system configuration of the compression processing system 1600 in the second phase
  • FIG. 17 shows the system configuration of the compression processing system 1600 in the third phase.
  • the compression processing system 1600 in the first phase includes an image pickup device 110, an analysis device 120, a data processing device 1610, and an image compression device 130.
  • the processing by the imaging device 110 and the image compression device 130 is the same as the processing by the imaging device 110 and the image compression device 130 described with reference to 11a or 11b of FIG. 11 in the second embodiment. The explanation will be omitted.
  • the analysis device 120 has a trained model that performs recognition processing.
  • the analysis device 120 performs recognition processing by inputting image data into the trained model, and outputs the recognition result. Further, the analysis device 120 acquires each compressed data output by the image compression device 130 performing compression processing on the image data at different compression levels (quantization values), and decodes each compressed data. Generate each decrypted data. Further, the analysis device 120 performs recognition processing by inputting each decoded data into the trained model, and outputs the recognition result.
  • the analysis device 120 generates an important feature map by analyzing the operation of the trained model at the time of recognition processing by using, for example, the error backpropagation method, and totals the degree of influence for each block.
  • the quantization value map (variable) in which the quantization value is set in each block is sequentially transmitted to the image compression device 130 to perform compression processing at different compression levels (quantization value). , Instruct the image compression device 130.
  • the analysis device 120 generates an aggregated value graph for each block based on the aggregated value of the influence degree of each block aggregated each time the recognition process is performed on each decoded data. Further, the analysis device 120 transmits each of the aggregated value graphs for each block to the data processing apparatus 1610 every time the aggregated value is updated.
  • the data processing device 1610 monitors the aggregated value graph transmitted from the analysis device 120 for each block, and determines whether or not the image data needs to be changed (for example, the size of the aggregated value of the aggregated value graph is predetermined). If the threshold is exceeded, it is determined that the image data needs to be changed). When the data processing device 1610 determines that it is not necessary to change the image data, the data processing device 1610 transmits the image data to the image compression device 130 without changing the image data.
  • the compression processing system 1600 in the second phase includes an image pickup device 110, an analysis device 120, a data processing device 1610, and an image compression device 130.
  • the processing by the imaging device 110 and the image compression device 130 is the same as the processing by the imaging device 110 and the image compression device 130 described with reference to 11a or 11b of FIG. 11 in the second embodiment. The explanation will be omitted. Further, since the processing by the analysis device 120 is the same as the processing by the analysis device 120 in the first phase, the description thereof will be omitted here.
  • the data processing device 1610 monitors the aggregated value graph transmitted from the analysis device 120 for each block, and determines whether or not the image data needs to be changed.
  • the data processing device 1610 determines that the image data needs to be changed, the data processing device 1610 changes the image data and transmits the changed image data to the image compression device 130.
  • the compression processing system 1600 in the third phase includes an analysis device 120, a data processing device 1610, and an image compression device 130.
  • the analysis device 120 determines the optimum compression level (quantization value) of each block based on the generated aggregated value graph, and generates a determined quantization value map. Further, the analysis device 120 transmits the generated determined quantization value map to the image compression device 130.
  • the data processing device 1610 transmits the changed image data to the image compression device 130.
  • the image compression device 130 performs compression processing on the changed image data using the determined quantization value map, and stores the compressed data in the storage device 150.
  • FIG. 18 is a third diagram showing an example of the functional configuration of the data processing device. Similar to the second embodiment, the data processing device 1610 has a data processing program installed, and when the program is executed, the data processing device 1610 functions as an input unit 1810 and a determination unit 1820. Further, the data processing device 1610 functions as an analysis unit 1230 and an image data change unit 1240.
  • the processing of the analysis unit 1230 and the image data change unit 1240 is the same as the processing of the analysis unit 1230 and the image data change unit 1240 of the data processing device 1110 in FIG. 12, so the description thereof is omitted here.
  • the input unit 1810 acquires image data from the analysis device 120. Further, when the determination unit 1820 notifies the determination unit 1810 of the determination result that the image data needs to be changed, the input unit 1810 notifies the analysis unit 1230 and the image data change unit 1240 of the acquired image data. In this case, the image data changing unit 1240 changes the image data based on the change information, and transmits the changed image data to the image compression device 130.
  • the input unit 1810 notifies the image data change unit 1240 of the acquired image data.
  • the image data changing unit 1240 transmits the image data to the image compression device 130 without changing the image data.
  • the determination unit 1820 monitors the aggregated value graph (an example of information related to the recognition accuracy of the image data) of each block transmitted from the analysis device 120, and determines whether or not the image data needs to be changed. When it is determined that the image data needs to be changed, the determination unit 1820 notifies the input unit 1810 of the determination result. On the other hand, when it is determined that the image data does not need to be changed, the determination unit 1820 notifies the input unit 1810 of the determination result.
  • the aggregated value graph an example of information related to the recognition accuracy of the image data
  • FIG. 19 is a third flowchart showing an example of the flow of image compression processing by the compression processing system.
  • step S1901 the input unit 310 of the analysis device 120 acquires image data.
  • step S1902 the quantization value setting unit 330 of the analysis device 120 transmits a quantization value map (variable) in which the minimum quantization value (Q 1) is set to the image compression device 130.
  • step S1903 the image compression device 130 performs compression processing on the image data using the transmitted quantization value map (variable) to generate compressed data.
  • step S1904 the input unit 310 of the analysis device 120 decodes the generated compressed data.
  • the CNN unit 320 of the analysis device 120 performs recognition processing on the decoded data.
  • step S1905 the important feature map generation unit 350 of the analysis device 120 generates an important feature map showing the degree of influence on the recognition result of each region based on the CNN part structure information.
  • step S1906 the aggregation unit 360 of the analysis device 120 aggregates the degree of influence of each region in block units based on the important feature map. Further, the aggregation unit 360 of the analysis device 120 stores the aggregation result in the aggregation result storage unit 390 in association with the current compression level (quantized value), and transmits the aggregation value graph to the data processing device 1610. ..
  • step S1907 the determination unit 1820 of the data processing device 1610 monitors the aggregated value graph of each block transmitted from the analysis device 120, and determines whether or not the image data needs to be changed.
  • step S1907 When it is determined in step S1907 that the image data needs to be changed (Yes in step S1907), the determination result is notified to the input unit 1810, and the process proceeds to step S1908.
  • step S1908 the input unit 1810 of the data processing device 1610 notifies the analysis unit 1230 and the image data change unit 1240 of the image data, and the analysis unit 1230 notifies the image data change unit 1240 of the change information. Further, the image data changing unit 1240 changes the image data based on the changed information, and transmits the changed image data to the image compression device 130.
  • the input unit 1810 of the data processing device 1610 notifies the analysis unit 1230 of the image data, and the analysis unit 1230 notifies the image data change unit 1240 of the score maximizing refined image. Further, the image data changing unit 1240 transmits the score maximizing refined image as the changed image data to the image compression device 130.
  • step S1907 when it is determined in step S1907 that the image data does not need to be changed (No in step S1907), the determination result is notified to the input unit 1810.
  • the input unit 1810 of the data processing device 1610 notifies the image data changing unit 1240 of the image data, and the image data changing unit 1240 transmits the image data to the image compression device 130 without changing the image data.
  • step S1909 the quantization value setting unit 330 of the analyzer 120 determines whether or not to set the next quantization value, and if it is determined that the next quantization value is to be set (Yes in step S1909). , Step S1910.
  • step S1910 the quantization value setting unit 330 of the analysis device 120 transmits the quantization value map (variable) in which the next quantization value is set to the image compression device 130, and then returns to step S1903.
  • step S1909 determines whether the next quantization value is set (if No in step S1909). If it is determined in step S1909 that the next quantization value is not set (if No in step S1909), the process proceeds to step S1911.
  • step S1911 the quantization value determination unit 370 of the analyzer 120 determines the quantization value in block units based on the aggregation value graph read from the aggregation result storage unit 390, and generates a quantization value map.
  • step S1912 the foreground determination unit 380 of the analyzer 120 maximizes the quantization value set in the background block among the generated quantization value maps, and generates a determined quantization value map.
  • step S1913 the image compression device 130 performs compression processing on the changed image data using the determined quantization value map, and stores the compressed data in the storage device 150.
  • the data processing apparatus displays the aggregated value graph of each block in the process of increasing the quantization value when generating the determined quantization value map. By monitoring, it is determined whether or not the image data needs to be changed. Further, the data processing apparatus according to the third embodiment changes the image data so as to maximize the score information when it is determined that the image data needs to be changed.
  • the recognition accuracy is improved even when the image data having low recognition accuracy is acquired. be able to.
  • the determined quantization value map is generated based on the changed image data, according to the third embodiment, the determined quantization value map in which a high quantization value is set can be generated.
  • the third embodiment it is possible to improve the compression level while improving the recognition accuracy as in the second embodiment. That is, according to the data processing apparatus according to the third embodiment, it is possible to realize a compression process suitable for the image recognition process by AI.
  • the fourth embodiment it is necessary to change the image data by confirming the recognition accuracy of the compressed data after the compression process is performed using the generated determined quantization value map. Is determined.
  • the fourth embodiment it is possible to improve the compression level while improving the recognition accuracy as in the third embodiment.
  • the fourth embodiment will be described focusing on the differences from each of the above embodiments.
  • the processing executed by the compression processing system is -The first phase in which a decision quantization value map is generated and compression processing is performed using the generated decision quantization value map, and -The second phase of checking the recognition accuracy of compressed data and changing the image data, -The changed image data can be compressed and roughly divided into a third phase in which the compressed data is stored.
  • 20a shows the system configuration of the compression processing system 2000 in the first phase
  • 20b shows the system configuration of the compression processing system in the second phase
  • FIG. 21 shows the system configuration of the compression processing system in the third phase.
  • the compression processing system 2000 in the first phase includes an image pickup device 110, an analysis device 120, and an image compression device 130. Since the processing by the imaging device 110 in the first phase is the same as the processing by the imaging device 110 described with reference to 1a of FIG. 1 in the first embodiment, the description thereof is omitted here.
  • processing by the analysis device 120 and the image compression device 130 in the first phase is the same as the processing by the analysis device 120 and the image compression device 130 described with reference to 11b of FIG. 11 in the second embodiment. , The description is omitted here.
  • the compression processing system 2000 in the second phase includes an analysis device 120, an image compression device 130, and a data processing device 2010.
  • the processing by the analysis device 120 and the image compression device 130 is the same as the processing by the analysis device 120 and the image compression device 130 described with reference to 11b of FIG. 11 in the second embodiment, and thus will be described here. Is omitted.
  • the data processing device 2010 decodes the compressed data transmitted from the image compression device 130, and performs recognition processing on the decoded data. Further, the data processing device 2010 determines whether or not the score information included in the recognition result satisfies a predetermined condition, and if it is determined that the score information does not satisfy the predetermined condition, the image data is maximized so that the score information is maximized. The changed image data is transmitted to the image compression device 130.
  • the compression processing system 2000 in the third phase includes an image compression device 130, a data processing device 2010, and a storage device 150.
  • the image compression device 130 in the third phase performs compression processing on the changed image data transmitted from the data processing device 2010 using the determined quantization value map, and compresses the compressed data. It is transmitted to the data processing device 2010.
  • the data processing device 2010 in the third phase decodes the compressed data transmitted from the image compression device 130 and performs recognition processing on the decoded data. Further, the data processing device 2010 determines whether or not the score information included in the recognition result satisfies a predetermined condition, and if it is determined that the predetermined condition is satisfied, the compressed data is stored in the storage device 150.
  • FIG. 22 is a fourth diagram showing an example of the functional configuration of the data processing device. Similar to the second embodiment, a data processing program is installed in the data processing device 2010, and when the program is executed, the data processing device 2010 has a decoding unit 2210, a CNN unit 1210, and a determination unit 1220. Functions as. Further, the data processing device 2010 functions as an analysis unit 1230 and an image data change unit 2240.
  • the CNN unit 1210, the determination unit 1220, and the analysis unit 1230 have the same functions as the CNN unit 1210, the determination unit 1220, and the analysis unit 1230 described with reference to FIG. 12 in the second embodiment. The explanation will be omitted.
  • the decoding unit 2210 decodes the compressed data transmitted from the image compression device 130 and generates the decrypted data. Further, the decoding unit 2210 notifies the CNN unit 1210 of the decoded data. Further, the decoding unit 2210 notifies the analysis unit 1230 of the decoded data in response to an instruction from the analysis unit 1230.
  • the image data changing unit 2240 is an example of the changing unit.
  • the image data change unit 2240 transmits the compressed data to the storage device 150.
  • the image data change unit 2240 changes the image data based on the notified change information and transmits the changed image data to the image compression device 130.
  • the image data changing unit 2240 transmits the changed image data to the image compression device 130.
  • FIG. 23 is a fourth flowchart showing an example of the flow of image compression processing by the compression processing system.
  • steps S1001 to S1007 are the same processes as steps S1001 to S1007 in FIG. 10, description thereof will be omitted, and here, the processes of steps S2301 to S2306 will be described.
  • step S2301 the image compression device 130 performs compression processing on the image data using the determined quantization value map to generate the compressed data.
  • step S2302 the decoding unit 2210 of the data processing device 2010 decodes the compressed data, and the CNN unit 1210 of the data processing device 2010 performs recognition processing on the decoded data to output the recognition result.
  • step S2303 the determination unit 1220 of the data processing device 2010 determines whether or not the image data needs to be changed by determining whether or not the score information included in the recognition result satisfies a predetermined condition.
  • step S2303 If it is determined in step S2303 that the predetermined condition is not satisfied (in the case of Yes in step S2303), it is determined that the image data needs to be changed, and the process proceeds to step S2304.
  • step S2304 the analysis unit 1230 of the data processing device 2010 generates change information for changing the image data so that the score information is maximized. Further, the image data changing unit 1240 of the data processing device 1110 changes the image data based on the generated change information, and transmits the changed image data to the image compression device 130.
  • the analysis unit 1230 of the data processing device 1110 generates a score maximizing refined image by changing the image data so as to maximize the score information, and notifies the image data changing unit 1240. Further, the image data changing unit 1240 of the data processing device 1110 transmits the score maximizing refined image as the changed image data to the image compression device 130.
  • step S2305 the image compression device 130 performs compression processing on the changed image data using the determined quantization value map to generate compressed data.
  • step S2303 determines whether the predetermined condition is satisfied (No in step S2303), it is determined that the image data does not need to be changed, and the process proceeds to step S2306 without changing the image data. ..
  • step S2306 the data processing device 2010 stores the compressed data in the storage device 150.
  • the data processing apparatus acquires the compressed data when the compression process is performed using the generated determined quantization value map, and the acquired compressed data.
  • the decrypted data obtained by decoding the data is recognized.
  • the data processing device determines whether or not the score information included in the recognition result satisfies a predetermined condition, and when it is determined that the predetermined condition is not satisfied, the score information is maximum. Change the image data so that it becomes.
  • the data processing apparatus stores the compressed data when the modified image data is compressed by using the determined quantization value map.
  • the recognition accuracy of the compressed data is confirmed, and when it is necessary to change the image data, the image data is changed. Therefore, according to the fourth embodiment, the compressed data having low recognition accuracy is output. Can be avoided. Thereby, according to the fourth embodiment, the recognition accuracy can be improved while improving the compression level. That is, according to the data processing apparatus according to the fourth embodiment, it is possible to realize a compression process suitable for the recognition process by AI.
  • the block is quantized. Described as maximizing the value.
  • the processing order between the process of generating the determined quantization value map and the process of maximizing the quantization value of the background block is not limited to this, and the process of maximizing the quantization value of the background block is performed. After that, a process for generating a determined quantization value map may be performed.
  • the process of maximizing the quantization value of the block is described, but the process of invalidating the image data of the background block (for example, a process of setting the pixel value to zero) may be performed.
  • low-pass filter processing such as blurring may be performed on the image data of the background block.
  • the image data referred to when the image compression device 130 performs the compression process on the image data is not particularly mentioned, but the image data to be referred to is a corrected quantization value.
  • the image data may be compressed using a map.
  • the image data to be referred to is image data that has been compressed using another quantization value map that produces the same degree of deterioration as when compression processing was performed using the corrected determined quantization value map. It may be.
  • a permissible range defined based on the recognition result for the image data is used as a predetermined second condition for determining whether or not to continue the process of increasing the quantization value.
  • the predetermined second condition is not limited to this.
  • an allowable range may be defined based on the recognition result for the compressed data when the compression process is performed at a predetermined compression level (quantization value).
  • the recognition result for the image data is used when defining the allowable range.
  • the information used when defining the permissible range is not limited to the recognition result for the image data, and for example, the annotation information given to the image data may be used.
  • the quantization value used when the image compression device 130 performs the compression processing has been described as being provided by the data processing device 140.
  • the data processing device 140 provides a weighting index for adjusting the quantization value used when the image compression device 130 performs compression processing, and the quantization value is based on the weighting index provided by the image compression device 130. May be adjusted.
  • the quantization value of each block of the map may be regarded as a weighting index of each block.
  • the quantization value of each block determined based on the algorithm for controlling the bit rate may be adjusted by using the weighting index.
  • the quantization value of each block fixedly set within a frame or over a plurality of frames may be adjusted by using a weighting index.
  • the image compression device 130 uses the image compression device 130 for a block having a large quantization value in the determination quantization value map.
  • Each quantized value determined based on the algorithm that controls the bit rate, or -Quantization value of each block set fixedly within a frame or across multiple frames, When adjusting in the direction of increasing, the strength of adjustment may be increased, and when adjusting in the direction of decreasing, the strength of adjustment may be decreased.
  • the image compression device 130 uses the image compression device 130 for a block having a large aggregated value. -Each quantized value determined based on the algorithm that controls the bit rate, or -Quantization value of each block set fixedly within a frame or across multiple frames, When adjusting in the direction of increasing, the strength of adjustment may be increased, and when adjusting in the direction of decreasing, the strength of adjustment may be decreased.
  • the quantization value adjusted by using the weighting index may be further changed according to other information.
  • Other information referred to here includes changes and transitions of values that affect recognition accuracy, such as score information, classification probability, error information, or object position information when compressed data is decoded and recognized. Is included.
  • the quantization value is changed so that the value that affects the recognition accuracy is maintained or improved, or is within a predetermined allowable range. And.
  • the corresponding image data or the image data acquired after the corresponding image data is compressed by using the changed quantization value. ..
  • the image compression device 130 performs compression processing using the changed quantization value on a plurality of image data including the corresponding image data and the image data acquired after the corresponding image data. And.
  • the number of objects included in the image data to be changed is not mentioned, but the number of objects included in the image data to be changed may be plural.
  • the data processing device may change the image data for each object so as to maximize the score information of each object, or for a plurality of objects so as to maximize the score information of the plurality of objects.
  • the image data may be changed collectively.
  • the score maximizing refined image data is generated when the image data is changed.
  • the decoded data or image data having high recognition accuracy may be used instead of generating the score maximizing refined image.
  • the process of generating the score maximizing refined image can be omitted.
  • the score maximizing refined image data is generated based on the image data or the decoded data.
  • the background block is determined and the image data or decoded data of the determined background block is invalidated before the score maximization refined image is generated, or image processing such as low-pass filter processing is performed. good.
  • the image data transmitted from the image pickup apparatus 110 is subjected to the compression process.
  • the target to be compressed is not limited to this, and for example, the image data transmitted from the image pickup apparatus 110 may be compressed to the image data resized to a predetermined size.
  • the size of each block is not particularly mentioned, but the size of each block may be a fixed size or a variable size. Further, in the case of a variable size, the size may be, for example, according to the magnitude of the quantization value.
  • Compression processing system 120 Analysis device 130: Image compression device 140: Data processing device 310: Input unit 320: CNN unit 330: Quantized value setting unit 340: Output unit 350: Important feature map generation unit 360: Aggregation unit 370 : Quantized value determination unit 380: Foreground determination unit 420: Aggregation result 810: Coding unit 820: Decoding unit 830: CNN unit 840: Quantization value correction unit 1100: Compression processing system 1110: Data processing device 1210: CNN unit 1220 : Judgment unit 1230: Analysis unit 1240: Image data change unit 1310: Refined image generation unit 1320: Important feature index map generation unit 1340: Specific unit 1350: Detailed analysis unit 1600: Compression processing system 1610: Data processing device 2000: Compression processing System 2010: Data processing device 2210: Decoding unit 2240: Image data changing unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Quality & Reliability (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
PCT/JP2020/003785 2020-01-31 2020-01-31 データ処理装置及びデータ処理プログラム Ceased WO2021152849A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/JP2020/003785 WO2021152849A1 (ja) 2020-01-31 2020-01-31 データ処理装置及びデータ処理プログラム
JP2021574421A JP7409400B2 (ja) 2020-01-31 2020-01-31 データ処理装置及びデータ処理プログラム
US17/838,321 US20220312019A1 (en) 2020-01-31 2022-06-13 Data processing device and computer-readable recording medium storing data processing program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/003785 WO2021152849A1 (ja) 2020-01-31 2020-01-31 データ処理装置及びデータ処理プログラム

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/838,321 Continuation US20220312019A1 (en) 2020-01-31 2022-06-13 Data processing device and computer-readable recording medium storing data processing program

Publications (1)

Publication Number Publication Date
WO2021152849A1 true WO2021152849A1 (ja) 2021-08-05

Family

ID=77079801

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/003785 Ceased WO2021152849A1 (ja) 2020-01-31 2020-01-31 データ処理装置及びデータ処理プログラム

Country Status (3)

Country Link
US (1) US20220312019A1 (https=)
JP (1) JP7409400B2 (https=)
WO (1) WO2021152849A1 (https=)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023032518A1 (en) * 2021-09-06 2023-03-09 Nec Corporation Data compression system and method of using
JPWO2023047516A1 (https=) * 2021-09-24 2023-03-30
WO2023181323A1 (ja) * 2022-03-25 2023-09-28 富士通株式会社 画像処理システム、画像処理装置、画像処理方法及び画像処理プログラム

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022130497A1 (ja) 2020-12-15 2022-06-23 富士通株式会社 解析装置、解析方法及び解析プログラム
WO2023286889A1 (ko) * 2021-07-15 2023-01-19 엘지전자 주식회사 A/v 수신 장치 및 무선 디스플레이 시스템
US12581108B2 (en) * 2024-06-24 2026-03-17 Advanced Micro Devices, Inc. Image sensor processing core for video pre-analysis

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003124815A (ja) * 2001-10-15 2003-04-25 Japan Science & Technology Corp 部分選択変換装置、部分選択変換方法及び部分選択変換プログラム
JP2016025494A (ja) * 2014-07-22 2016-02-08 ルネサスエレクトロニクス株式会社 画像受信装置、画像伝送システムおよび画像受信方法
JP2016046707A (ja) * 2014-08-25 2016-04-04 ルネサスエレクトロニクス株式会社 画像通信装置、画像送信装置および画像受信装置
JP2017073755A (ja) * 2015-10-09 2017-04-13 キヤノン株式会社 符号化装置及びその制御方法、プログラム並びに記憶媒体

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003124815A (ja) * 2001-10-15 2003-04-25 Japan Science & Technology Corp 部分選択変換装置、部分選択変換方法及び部分選択変換プログラム
JP2016025494A (ja) * 2014-07-22 2016-02-08 ルネサスエレクトロニクス株式会社 画像受信装置、画像伝送システムおよび画像受信方法
JP2016046707A (ja) * 2014-08-25 2016-04-04 ルネサスエレクトロニクス株式会社 画像通信装置、画像送信装置および画像受信装置
JP2017073755A (ja) * 2015-10-09 2017-04-13 キヤノン株式会社 符号化装置及びその制御方法、プログラム並びに記憶媒体

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023032518A1 (en) * 2021-09-06 2023-03-09 Nec Corporation Data compression system and method of using
US11764806B2 (en) 2021-09-06 2023-09-19 Nec Corporation Data compression system and method of using
JP2024530346A (ja) * 2021-09-06 2024-08-16 日本電気株式会社 データ圧縮システム及び使用方法
JP7616483B2 (ja) 2021-09-06 2025-01-17 日本電気株式会社 データ圧縮システム及び使用方法
JPWO2023047516A1 (https=) * 2021-09-24 2023-03-30
WO2023047516A1 (ja) * 2021-09-24 2023-03-30 富士通株式会社 画像処理システム、符号化装置、符号化方法及び符号化プログラム
JP7670149B2 (ja) 2021-09-24 2025-04-30 富士通株式会社 画像処理システム、符号化装置、符号化方法及び符号化プログラム
WO2023181323A1 (ja) * 2022-03-25 2023-09-28 富士通株式会社 画像処理システム、画像処理装置、画像処理方法及び画像処理プログラム
JPWO2023181323A1 (https=) * 2022-03-25 2023-09-28
JP7700956B2 (ja) 2022-03-25 2025-07-01 富士通株式会社 画像処理システム、画像処理装置、画像処理方法及び画像処理プログラム

Also Published As

Publication number Publication date
US20220312019A1 (en) 2022-09-29
JP7409400B2 (ja) 2024-01-09
JPWO2021152849A1 (https=) 2021-08-05

Similar Documents

Publication Publication Date Title
JP7409400B2 (ja) データ処理装置及びデータ処理プログラム
US20220284632A1 (en) Analysis device and computer-readable recording medium storing analysis program
US11831909B2 (en) Learned B-frame coding using P-frame coding system
US7843995B2 (en) Temporal and spatial analysis of a video macroblock
EP4425918A2 (en) Method and apparatus for filtering with mode-aware deep learning
US10536696B2 (en) Image encoding device and image encoding method
US20180124431A1 (en) In-loop post filtering for video encoding and decoding
JP2022078735A (ja) 画像処理装置、画像処理プログラム、画像認識装置、画像認識プログラム及び画像認識システム
US20210266570A1 (en) Region of interest quality controllable video coding techniques
CN114268792A (zh) 视频转码方案的确定方法及装置和视频转码方法及装置
CN112399177A (zh) 一种视频编码方法、装置、计算机设备及存储介质
US20230262236A1 (en) Analysis device, analysis method, and computer-readable recording medium storing analysis program
US12236669B2 (en) Image processing system, image processing method, and storage medium
US12170786B2 (en) Device and method for encoding and decoding image using AI
KR20250155015A (ko) 효율적인 워핑 기반 뉴럴 비디오 코덱
Moniruzzaman et al. Structure‐Aware Image Translation‐Based Long Future Prediction for Enhancement of Ground Robotic Vehicle Teleoperation
US20210306641A1 (en) Transcoder target bitrate prediction techniques
WO2022036678A1 (en) Multi-level region-of-interest quality controllable video coding techniques
US20230014220A1 (en) Image processing system, image processing device, and computer-readable recording medium storing image processing program
US20240046527A1 (en) End-to-end optimization of adaptive spatial resampling towards machine vision
US20140270538A1 (en) Moving image processing apparatus, moving image processing method, and computer product
EP4704418A1 (en) Method and apparatus using intra prediction fusion
Lin et al. Spatial-temporal feature guided adaptive rate control for screen content videos
Antony et al. Selective intra prediction in HEVC planar and angular modes for efficient near-lossless video compression
KR20230120854A (ko) 전자 장치의 영상 부호화 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20916667

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021574421

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20916667

Country of ref document: EP

Kind code of ref document: A1