WO2019047628A1 - 海量图片处理方法、装置、电子设备及存储介质 - Google Patents

海量图片处理方法、装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2019047628A1
WO2019047628A1 PCT/CN2018/097010 CN2018097010W WO2019047628A1 WO 2019047628 A1 WO2019047628 A1 WO 2019047628A1 CN 2018097010 W CN2018097010 W CN 2018097010W WO 2019047628 A1 WO2019047628 A1 WO 2019047628A1
Authority
WO
WIPO (PCT)
Prior art keywords
picture
layer
index
bit
layer data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/097010
Other languages
English (en)
French (fr)
Inventor
杨丽敏
刘诏
薛韬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Jingdong Shangke Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to US16/642,095 priority Critical patent/US11395010B2/en
Publication of WO2019047628A1 publication Critical patent/WO2019047628A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/54—Browsing; Visualisation therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/51—Indexing; Data structures therefor; Storage structures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/55—Clustering; Classification
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/39—Control of the bit-mapped memory
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
    • H04N19/39—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability involving multiple description coding [MDC], i.e. with separate layers being structured as independently decodable descriptions of input picture data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/46—Embedding additional information in the video signal during the compression process
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H04N19/93—Run-length coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
    • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
    • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
    • H04N19/184—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being bits, e.g. of the compressed video stream

Definitions

  • the present disclosure relates to the field of data processing technologies, and in particular, to a mass picture processing method, a massive picture processing apparatus, an electronic device, and a computer readable storage medium.
  • the internal system will store a large number of images, such as e-commerce websites will have a large number of merchandise pictures and product details pictures. Since the processing of massive pictures and data consumes a large amount of storage medium and transmission bandwidth, it is necessary to process massive amounts of data to improve storage and transmission speed.
  • the processing methods for massive pictures in the related art are mostly considered from the aspects of improving picture storage, reading speed, and mass information management.
  • the processing method of massive pictures because the image storage amount is very large, it will occupy a large amount of storage space and network transmission bandwidth, resulting in a problem of slow transmission speed.
  • a method for processing a massive image including:
  • the picture bit layer data corresponding to each of the channels layered by bit and generating a layer index record file; wherein the inverted index structure includes multiple layer indexes;
  • All the layer indexes of the picture are decoded according to the layer index record file, and the picture is synthesized according to the decoding result.
  • performing bitwise layering on each of the matrices includes:
  • the method further includes:
  • Correlation calculation is performed on a matrix corresponding to each layer of each channel, and a difference matrix corresponding to the correlation is obtained;
  • storing the picture bit layer data corresponding to each of the channels layered by bit according to an inverted index structure includes:
  • detecting whether the picture bit layer data is completely matched includes:
  • the index value is recalculated when it is detected that the picture bit layer data does not completely match.
  • decoding all of the layer indexes of the picture includes:
  • the method further includes:
  • the picture generated according to the decoding result is verified by the index value.
  • a mass image processing apparatus including:
  • a layering module configured to obtain a matrix corresponding to multiple channels of each picture in the massive picture, and perform bitwise layering on each of the matrixes;
  • a storage module configured to store, according to an inverted index structure, image layer data corresponding to each of the channels layered by bit, and generate a layer index record file; wherein the inverted index structure includes multiple maps Layer index
  • a decoding module configured to decode all the layer indexes of the picture according to the layer index record file, and synthesize the picture according to the decoding result.
  • a computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the massive image processing method of any of the above.
  • an electronic device including:
  • a memory for storing executable instructions of the processor
  • the processor is configured to perform the massive image processing method according to any one of the above items by executing the executable instruction.
  • FIG. 1 is a schematic diagram showing a mass picture processing method in an exemplary embodiment of the present disclosure
  • FIG. 2 is a schematic diagram showing three channels of a picture in an exemplary embodiment of the present disclosure
  • FIG. 3 is a schematic diagram showing the correlation of three channel decomposition matrices in an exemplary embodiment of the present disclosure
  • FIG. 4 is a schematic diagram showing a layer 8 binary map of a picture in an exemplary embodiment of the present disclosure
  • FIG. 5 is a block diagram schematically showing a mass image processing apparatus in an exemplary embodiment of the present disclosure
  • FIG. 6 schematically illustrates an electronic device in an exemplary embodiment of the present disclosure
  • FIG. 7 schematically shows a program product in an exemplary embodiment of the present disclosure.
  • the massive image processing method in the related art stores, for example, structured data in Hbase, and the unstructured picture data is directly stored in the Hadoop distributed file system, and the storage address of the image is stored in Hbase. In this way, the transmission speed is slow.
  • the massive image processing method may include the following steps:
  • Step S110 acquiring a matrix corresponding to multiple channels of each picture in the massive picture, and performing bitwise layering on each of the matrices;
  • Step S120 storing image bit layer data corresponding to each of the channels layered by bit according to an inverted index structure, and generating a layer index record file;
  • Step S130 Decode all the layer indexes of the picture according to the layer index record file, and synthesize the picture according to the decoding result.
  • the picture bit layer data is obtained by layer-by-bit layering the matrix of each channel of the picture, and the picture bit layer data is combined and stored according to the inverted index structure.
  • the picture bit layer data is combined and stored, which reduces the transmission amount of the picture, thereby improving the picture transmission speed and efficiency.
  • step S110 a matrix corresponding to a plurality of channels of each picture in the massive picture is acquired, and each of the matrices is layered in a bit.
  • the massive picture may include, for example, a product picture, a product detail picture, and the like in an e-commerce website, and may also include a large number of pictures on other websites.
  • Each picture is composed of channels.
  • the grayscale image is composed of one channel.
  • the 32-bit RGB images are composed of three channels: R, G, and B.
  • R, G, and B 32-bit RGB images are used in this example.
  • the matrix corresponding to multiple channels of each picture may be a partial matrix or a complete matrix, and all the matrices corresponding to each channel may be obtained by loop. It should be added that each channel is composed of a matrix of decimal numbers, that is, each element in the matrix is a decimal element.
  • the original picture can be decomposed into an R channel, a G channel, and a B channel, and the matrix corresponding to each channel corresponding to any picture can be layerwise layered.
  • performing bitwise layering on each of the matrices may include: converting each decimal element in a matrix corresponding to each channel into a binary element, and corresponding to each channel according to the binary bit.
  • the matrix is layered in bits.
  • each element in the matrix corresponding to each channel is a decimal element, and all the decimal elements can be converted into corresponding binary elements.
  • the conversion process can be performed by a program.
  • Each channel can correspond to a plurality of different matrices, and each element in each matrix can correspond to one pixel.
  • the elements in all the matrices corresponding to each channel can be converted to perform bitwise layering. For example, a specific step of bitwise layering can be illustrated by taking a partial matrix of R channels as an example.
  • Bitwise layering can be understood as layering each binary element in a matrix according to the number of bits of a binary element. For example, if the number of binary elements is 8, then each binary element can be decomposed into 8 matrices.
  • equation (1) For example, a small part of the matrix of the R channel is shown in equation (1):
  • Each element of the R-channel matrix can be layered in binary bits, and each channel can be decomposed into 8 matrices according to binary bits.
  • decimal 55 can be converted to binary 00110111, then element 55 can be decomposed into [0 0 1 1 0 1 1]; decimal 167 can be converted to binary 10100111, then element 167 can be correspondingly decomposed into [1 0 1 0 0 1 1 1].
  • the partial matrix corresponding to the R channel can be decomposed into eight matrices as shown in equation (3). Since all the elements of the eight matrices only contain 0 and 1, the matrix corresponding to each channel can be decomposed into 8 according to the binary bit. 01 matrix.
  • the matrix corresponding to the G channel and the B channel may be layered according to the binary bits, thereby obtaining eight 01 matrices G1, G2, G3, G4, G5 corresponding to the G channel.
  • the method may further include: performing correlation calculation on a matrix corresponding to each layer of each of the channels, and acquiring related a difference matrix corresponding to the sex; performing run length coding on the difference matrix, and acquiring a plurality of the picture bit layer data corresponding to each layer of each of the channels.
  • the pictures of the three channels that are decomposed can be first grayed out, as shown in FIG. 2 .
  • the grayscale maps corresponding to the respective channels have similarities, so it can be predicted that there is a strong correlation between the corresponding layers of the three sets of matrixes R1-8, G1-8 and B1-8 which are decomposed.
  • a correlation calculation is performed on the matrix corresponding to each layer of each of the channels. Specific steps are as follows:
  • the correspondence between the three sets of decomposition matrices is as shown in FIG. 3. It can be seen from the corresponding relationship diagram that the decomposition matrices of the same layer of the three channels have a one-to-one correspondence, for example, R1 and G1 respectively. B1 corresponds to; R8 corresponds to G8 and B8, respectively.
  • the correlation can be represented by a difference matrix, which can be done using an exclusive OR operation.
  • the decomposition matrix of each layer of the G channel and the B channel can be represented by an R matrix corresponding to the decomposition matrix corresponding to the layer in combination with the difference matrix.
  • the three channels also have strong correlation between the binary pictures corresponding to the decomposition matrix of the eighth layer.
  • the binary image corresponding to R8 can be regarded as the binary image corresponding to G8.
  • the binary image corresponding to B8 is combined.
  • the decomposition matrices R1, G1, and B1 of the first layer of the three channels can be as shown in the formula (4).
  • both G1 and B1 can be represented by the form of R1 plus a difference matrix, such as shown in equation (5), and the difference matrix can be completed by an exclusive OR operation.
  • the difference matrices G1' and B1' are both sparse matrices, at which point R1, G1' and B1' can be recorded.
  • the correlation calculation does not require the use of the R channel, that is to say, in addition to this, the decomposition matrix of the same layer of the three channels can also be described by the G channel and the difference matrix R' and B'. Correlation between the two; the correlation between the decomposition matrices of the same layer of the three channels can also be described by the B channel and the difference matrices R' and G'.
  • the difference matrix may be subjected to Run Length Encoding (RLE) to obtain a plurality of picture bit layer data corresponding to each layer of each of the channels.
  • RLE Run Length Encoding
  • the run length encoding method is a lossless compression method that can replace adjacent pixels of the same color value in one scan line with a count value and color values of those pixels. For example: aaabccccccddeee, you can use 3a1b6c2d3e instead.
  • the string JJJJJJAAAAVVVVAAAAAA can be encoded in the form of 6J4A4V6A, where "6J” means 6 characters J, "4A” means 4 characters A, and stroke length encoding can avoid a lot of redundant information, making pictures The process is simpler and faster.
  • the difference matrix may be run length encoded to obtain a plurality of picture bit layer data corresponding to each layer of each of the channels, and the picture bit layer data is in one-to-one correspondence with the matrix.
  • the picture bit layer data may be a data sequence.
  • each sparse matrix may be run length encoded to obtain 24 data sequences and the pictures may be fully expressed by the 24 data sequences.
  • the data sequence obtained by compression-coding the matrix B1' is 01410.
  • the 24 data sequences can be expressed as:
  • step S120 the picture bit layer data corresponding to each of the channels layered by bit is stored according to an inverted index structure, and a layer index record file is generated.
  • an inverted index structure may be first established, and the inverted index structure may determine the position of the record according to the attribute value in an actual application.
  • the inverted index structure may be an index table, the first column of the index table may be used to store the picture bit layer data obtained by the run length encoding; the second column may be used to store the index value calculated according to the picture bit layer data.
  • the index value may include a HashCode index value or an MD5 index value; the third column of the index table may be used to store a layer index. If the picture is A, B, C, D, the inverted index structure can be as shown in Table 1.
  • the layer index record file may be used to record a process of storing all picture bit layer data according to the inverted index structure.
  • the storing the bit layer data corresponding to each of the channels in the bit-by-bit layer according to an inverted index structure may include: calculating an index value of the picture bit layer data, and Locating an index row matching the index value in the inverted index structure; adding a row layer index record in the inverted index structure when the index row matching the index value is not found; When the index row matching the index value is reached, it is detected whether the picture bit layer data is completely matched.
  • the index value corresponding to each picture bit layer data may be first calculated, that is, the data value stored in the data sequence corresponding to each layer of each channel, and the index value may include a HashCode index value or an MD5 index value, for example,
  • the function loop calculates and obtains the index value of the data sequence corresponding to each layer of each channel.
  • the picture may be hierarchically layered according to the foregoing steps to obtain 24 picture bit layer data. Take the decomposition of picture A as an example:
  • an index row corresponding to or matching the calculated index value can be found in the above index table. If the index row matching the index value is not found, a row layer index record may be added to the inverted index structure. Referring to Table 1, for example, if the currently calculated graph A-R3 has an index value of 486dgf4dfg4s3246, and does not match all the index rows in Table 1, the data sequence, the index value, and the corresponding layer index may be added to In the index table. When the index row matching the index value is found, it may be detected whether the picture bit layer data is completely matched.
  • the index value of the second layer R channel of the currently calculated picture A is gfGREdsfsdf89fds, and it can continue to detect whether the data sequence used for calculation and the data sequence stored in the index table completely match. Judging by judging each element in the matrix corresponding to the data sequence.
  • the detecting whether the picture bit layer data is completely matched may include: adding, when the picture bit layer data is completely matched, adding a mark corresponding to the layer to the inverted index structure; The index value is recalculated when it is detected that the picture bit layer data does not completely match.
  • the calculation when it is detected that the currently calculated picture bit layer data does not completely match or completely does not match the picture bit layer data stored in the index table, in order to improve the accuracy of the search, the calculation may be recalculated.
  • the index value is used.
  • the channel and the number of layers of the picture corresponding to the index value may be marked in the index table.
  • the calculated index A of the second layer R channel of the picture A is gfGREdsfsdf89fds, and each element in the matrix corresponding to the calculated data sequence is the same as each element of the data sequence stored in the index table, so
  • the map A-R2 corresponding to the second layer R channel of the picture A is recorded in the layer index column of the index table.
  • step S130 all the layer indexes of the picture are decoded according to the layer index record file, and the picture is generated according to the decoding result.
  • the layer index record file generated in step S120 may be decoded, and the picture generated according to the decoding result is a bit-layered picture.
  • the decoding process is an inverse process of the encoding process, and therefore, the encoding method and the decoding method can correspond to each other.
  • all the layer indexes stored in the index table corresponding to a certain picture may be decoded to obtain the original picture data corresponding to the three channels of the picture, thereby obtaining the original picture.
  • the decoding of all the layer indexes of the picture may include: recording, according to the layer index, all the layer indexes of the file decoding picture, and searching according to the layer index. And compressing the picture bit layer data corresponding to the picture; performing run length decoding on the compressed picture bit layer data, and acquiring an uncompressed original picture bit layer data matrix corresponding to the picture; The original picture bit layer data matrix is subjected to an exclusive OR operation, and an un-layered original matrix corresponding to the plurality of channels of the picture is obtained and the picture is synthesized according to the original matrix.
  • the raw data that can represent the picture can be obtained layer by layer from the lowest level of data.
  • all layer indexes of the file decoded picture may be recorded according to the layer index, for example, all layer indexes corresponding to the image A are obtained: FIG. A-R1-8, FIG. A-G1-8, and FIG.
  • searching, according to the layer index, the compressed picture bit layer data corresponding to the layer that is, searching a data sequence corresponding to all layer indexes of the picture A in the index table, for example, obtaining data.
  • Sequence 01410 it should be noted that the data sequence obtained here is a data sequence compressed by run length encoding.
  • run length decoding may be performed on the acquired compressed data sequence to obtain an original data matrix that is not encoded and compressed.
  • the run length decoding may be performed on the data sequence 01410 to obtain a corresponding original matrix B1'.
  • the obtained 24 data sequence cycles may be decoded to obtain uncompressed 24 original picture bit layer data matrices:
  • the difference matrix G1' and B1' in the obtained 24 original picture bit layer data matrices can be obtained by an exclusive OR operation
  • the difference matrix G1' and B1' can be XORed again to restore the unresolved Original matrix after any processing
  • the matrix corresponding to the first layer to the eighth layer of the three channels R, G, and B respectively can be used to represent three channels of RGB, and finally the original original picture can be synthesized through three channels.
  • the method may further include: performing, by using the index value, the picture generated according to the decoding result.
  • the picture generated after the decoding may be verified to determine whether the picture is consistent with the original bit-layered picture; of course, the verification may not be performed.
  • the specific verification process can be completed through the program, and will not be described here.
  • the massive pictures can be processed by cyclically combining the processes of bitwise layering, code storage, and decoding in this example.
  • the pictures are decomposed layer by layer and combined and stored in layer dimensions, which reduces the amount of image storage and transmission, saves storage space, and improves transmission. effectiveness.
  • the apparatus 500 may include:
  • the layering module 501 can be configured to obtain a matrix corresponding to multiple channels of each picture in the massive picture, and perform bitwise layering on each of the matrices;
  • the storage module 502 is configured to store, according to an inverted index structure, the picture bit layer data corresponding to each of the channels layered by bit, and generate a layer index record file; wherein the inverted index structure includes multiple Layer index;
  • the decoding module 503 is configured to decode all the layer indexes of the picture according to the layer index record file, and synthesize the picture according to the decoding result.
  • modules or units of equipment for action execution are mentioned in the detailed description above, such division is not mandatory. Indeed, in accordance with embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of one of the modules or units described above may be further divided into multiple modules or units.
  • an electronic device capable of implementing the above method is also provided.
  • FIG. 6 An electronic device 600 according to such an embodiment of the present disclosure is described below with reference to FIG. 6 is merely an example and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.
  • electronic device 600 is embodied in the form of a general purpose computing device.
  • the components of the electronic device 600 may include, but are not limited to, the at least one processing unit 610, the at least one storage unit 620, and a bus 630 that connects different system components (including the storage unit 620 and the processing unit 610).
  • the storage unit stores program code, which can be executed by the processing unit 610, such that the processing unit 610 performs various exemplary embodiments according to the present disclosure described in the "Exemplary Method" section of the present specification.
  • the processing unit 610 can perform the above steps.
  • the storage unit 620 can include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and/or a cache storage unit 6202, and can further include a read only storage unit (ROM) 6203.
  • RAM random access storage unit
  • ROM read only storage unit
  • the storage unit 620 can also include a program/utility 6204 having a set (at least one) of the program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more applications, other program modules, and program data, Implementations of the network environment may be included in each or some of these examples.
  • Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local area using any of a variety of bus structures. bus.
  • the electronic device 600 can also communicate with one or more external devices 700 (eg, a keyboard, pointing device, Bluetooth device, etc.), and can also communicate with one or more devices that enable the user to interact with the electronic device 600, and/or with The electronic device 600 is enabled to communicate with any device (e.g., router, modem, etc.) that is in communication with one or more other computing devices. This communication can take place via an input/output (I/O) interface 650. Also, electronic device 600 can communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet) through network adapter 660. As shown, network adapter 660 communicates with other modules of electronic device 600 via bus 630.
  • network adapter 660 communicates with other modules of electronic device 600 via bus 630.
  • the technical solution according to an embodiment of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network.
  • a non-volatile storage medium which may be a CD-ROM, a USB flash drive, a mobile hard disk, etc.
  • a number of instructions are included to cause a computing device (which may be a personal computer, server, terminal device, or network device, etc.) to perform a method in accordance with an embodiment of the present disclosure.
  • a computer readable storage medium having stored thereon a program product capable of implementing the above method of the present specification.
  • various aspects of the present disclosure may also be embodied in the form of a program product comprising program code for causing said program product to run on a terminal device The terminal device performs the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Method" section of the present specification.
  • a program product 800 for implementing the above method which may employ a portable compact disk read only memory (CD-ROM) and includes program code, and may be at a terminal device, is illustrated in accordance with an embodiment of the present disclosure.
  • CD-ROM portable compact disk read only memory
  • the program product of the present disclosure is not limited thereto, and in this document, the readable storage medium may be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.
  • the program product can employ any combination of one or more readable media.
  • the readable medium can be a readable signal medium or a readable storage medium.
  • the readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive lists) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.
  • the computer readable signal medium may comprise a data signal that is propagated in the baseband or as part of a carrier, carrying readable program code. Such propagated data signals can take a variety of forms including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the readable signal medium can also be any readable medium other than a readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
  • Program code embodied on a readable medium can be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.
  • Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language, such as Java, C++, etc., including conventional procedural Programming language—such as the "C" language or a similar programming language.
  • the program code can execute entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the remote computing device on the user computing device, or entirely on the remote computing device or server. Execute on.
  • the remote computing device can be connected to the user computing device via any kind of network, including a local area network (LAN) or wide area network (WAN), or can be connected to an external computing device (eg, provided using an Internet service) Businesses are connected via the Internet).
  • LAN local area network
  • WAN wide area network
  • Businesses are connected via the Internet.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • General Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Image Processing (AREA)

Abstract

本公开提供了一种海量图片处理方法、海量图片处理装置、电子设备以及计算机可读存储介质，涉及数据处理技术领域，该方法包括：获取海量图片中各图片的多个通道对应的矩阵，并对各所述矩阵进行按位分层；根据一倒排索引结构存储按位分层后的各所述通道对应的图片位层数据，并生成一图层索引记录文件；根据所述图层索引记录文件对所述图片的所有所述图层索引进行解码，并根据解码结果合成所述图片。该方法可以减少图片存储量，提高图片传输效率。

Description

海量图片处理方法、装置、电子设备及存储介质 技术领域
本公开涉及数据处理技术领域，具体而言，涉及一种海量图片处理方法、海量图片处理装置、电子设备以及计算机可读存储介质。
背景技术
随着电子商务的迅速发展，对于各大网站平台而言，其内部系统会存储海量图片，例如电商网站中会存在海量商品图片以及商品详情图片等。由于海量图片以及数据的处理会消耗大量的存储介质及传输带宽，因此有必要对海量数据进行处理以提高存储和传输速度。
相关技术中对海量图片的处理方式，大多从提高图片的存储、读取速度以及海量信息管理方面考虑。通过这种方式对海量图片的处理方法，由于图片存储量非常大，因此会占用大量存储空间以及网络传输带宽，导致传输速度慢的问题。
需要说明的是，在上述背景技术部分公开的信息仅用于加强对本公开的背景的理解，因此可以包括不构成对本领域普通技术人员已知的现有技术的信息。
发明内容
本公开的目的在于提供一种海量图片处理方法、海量图片处理装置、电子设备以及计算机可读存储介质，进而至少在一定程度上克服由于相关技术的限制和缺陷而导致的一个或者多个问题。
本公开的其他特性和优点将通过下面的详细描述变得显然，或部分地通过本公开的实践而习得。
根据本公开的第一方面，提供一种海量图片处理方法，包括：
获取海量图片中各图片的多个通道对应的矩阵，并对各所述矩阵进行按位分层；
根据一倒排索引结构存储按位分层后的各所述通道对应的图片位层数据，并生成一图层索引记录文件；其中，所述倒排索引结构包括多个图层索引；
根据所述图层索引记录文件对所述图片的所有所述图层索引进行解码，并根据解码结果合成所述图片。
在本公开的一种示例性实施例中，对各所述矩阵进行按位分层包括：
将各通道对应的矩阵中的各十进制元素转化为二进制元素，并根据二进制位对各所述通道对应的矩阵进行按位分层。
在本公开的一种示例性实施例中，对各所述矩阵进行按位分层后，所述方法还包括：
对各所述通道每一层对应的矩阵进行相关性计算，并获取与所述相关性对应的差异矩阵；
对所述差异矩阵进行行程长度编码，获取与各所述通道每一层对应的多个所述图片位层数据。
在本公开的一种示例性实施例中，根据一倒排索引结构存储按位分层后的各所述通道对应的图片位层数据包括：
计算所述图片位层数据的索引值，并在所述倒排索引结构中查找与所述索引值匹配的索引行；
在未查找到与所述索引值匹配的所述索引行时，在所述倒排索引结构中增加一行图层索引记录；
在查找到与所述索引值匹配的所述索引行时，检测所述图片位层数据是否完全匹配。
在本公开的一种示例性实施例中，检测所述图片位层数据是否完全匹配包括：
在检测所述图片位层数据完全匹配时，将图层对应的标记添加至所述倒排索引结构中；
在检测所述图片位层数据不完全匹配时，重新计算所述索引值。
在本公开的一种示例性实施例中，对所述图片的所有所述图层索引进行解码包括：
根据所述图层索引记录文件解码所述图片的所有所述图层索引，并根据所述图层索引查找压缩后的与所述图片对应的所述图片位层数据；
对压缩后的所述图片位层数据进行行程长度解码，获取未压缩的与所述图片对应的原始图片位层数据矩阵；
对所述原始图片位层数据矩阵进行异或运算，获取未分层的与所述图片的多个通道对应的原矩阵并根据所述原矩阵合成所述图片。
在本公开的一种示例性实施例中，所述方法还包括：
通过所述索引值对根据解码结果生成的所述图片进行校验。
根据本公开的一个方面，提供一种海量图片处理装置，包括：
分层模块，用于获取海量图片中各图片的多个通道对应的矩阵，并对各所述矩阵进 行按位分层；
存储模块，用于根据一倒排索引结构存储按位分层后的各所述通道对应的图片位层数据，并生成一图层索引记录文件；其中，所述倒排索引结构包括多个图层索引；
解码模块，用于根据所述图层索引记录文件对所述图片的所有所述图层索引进行解码，并根据解码结果合成所述图片。
根据本公开的一个方面，提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述任意一项所述的海量图片处理方法。
根据本公开的一个方面，提供一种电子设备，包括：
处理器；以及
存储器，用于存储所述处理器的可执行指令；
其中，所述处理器配置为经由执行所述可执行指令来执行上述任意一项所述的海量图片处理方法。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本公开。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1示意性示出本公开示例性实施例中一种海量图片处理方法的示意图；
图2示意性示出本公开示例性实施例中一种图片的三个通道示意图；
图3示意性示出本公开示例性实施例中三个通道分解矩阵的相关性示意图；
图4示意性示出本公开示例性实施例中图片第8层二值图示意图；
图5示意性示出本公开示例性实施例中一种海量图片处理装置的框图；
图6示意性示出本公开示例性实施例中一种电子设备；
图7示意性示出本公开示例性实施例中一种程序产品。
具体实施方式
现在将参考附图更全面地描述示例实施方式。然而，示例实施方式能够以多种形式实施，且不应被理解为限于在此阐述的范例；相反，提供这些实施方式使得本公开将更加全面和完整，并将示例实施方式的构思全面地传达给本领域的技术人员。所描述的特 征、结构或特性可以以任何合适的方式结合在一个或更多实施方式中。在下面的描述中，提供许多具体细节从而给出对本公开的实施方式的充分理解。然而，本领域技术人员将意识到，可以实践本公开的技术方案而省略所述特定细节中的一个或更多，或者可以采用其它的方法、组元、装置、步骤等。在其它情况下，不详细示出或描述公知技术方案以避免喧宾夺主而使得本公开的各方面变得模糊。
此外，附图仅为本公开的示意性图解，并非一定是按比例绘制。图中相同的附图标记表示相同或类似的部分，因而将省略对它们的重复描述。附图中所示的一些方框图是功能实体，不一定必须与物理或逻辑上独立的实体相对应。可以采用软件形式来实现这些功能实体，或在一个或多个硬件模块或集成电路中实现这些功能实体，或在不同网络和/或处理器装置和/或微控制器装置中实现这些功能实体。
相关技术中的海量图片处理方式例如将结构化数据存储到Hbase，非结构化的图片数据直接存储到Hadoop分布式文件系统，将图片的存储地址存储到Hbase中。通过这种方式导致传输速度慢。
本示例实施方式中，首先提供了一种海量图片处理方法，参考图1所示，所述海量图片处理方法可以包括以下步骤：
步骤S110，获取海量图片中各图片的多个通道对应的矩阵，并对各所述矩阵进行按位分层；
步骤S120，根据一倒排索引结构存储按位分层后的各所述通道对应的图片位层数据，并生成一图层索引记录文件；
步骤S130，根据所述图层索引记录文件对所述图片的所有所述图层索引进行解码，并根据解码结果合成所述图片。
在本公开示例性实施例提供的海量图片处理方法中，通过将图片各通道的矩阵进行按位分层得到图片位层数据，并根据倒排索引结构将图片位层数据进行合并存储，一方面，通过将图片各通道的矩阵进行按位分层，避免了各图片之间重复信息的冗余存储，减少了图片存储量，节约了存储空间；另一方面，通过根据倒排索引结构将图片位层数据进行合并存储，减少了图片的传输量，进而提高了图片传输速度和效率。
接下来，参考图1至图5对本示例实施方式中提供的海量图片处理方法中的各个步骤进行具体说明。
在步骤S110中，获取海量图片中各图片的多个通道对应的矩阵，并对各所述矩阵进行按位分层。
本示例性实施例中，海量图片例如可以包括电商网站中的商品图片、商品详情图片等，也可以包括其它网站上的海量图片。每张图片均是由通道组成，例如灰度图由一个通道组成，32位的RGB图像均由R、G、B三个通道组成，参考图2所示，本示例中以32位的RGB图像为例进行说明。各图片的多个通道对应的矩阵可以为部分矩阵，也可 以为完整矩阵，可以通过循环获取每个通道对应的所有矩阵。需要补充的是，每个通道都是由十进制数字矩阵组成，即矩阵中的每个元素均为十进制元素。本示例中，参考图2所示，可以将其中的原图分解为R通道、G通道和B通道，可以对任意一张图片对应的各个通道对应的矩阵进行按位分层。
具体而言，本示例实施方式中，对各所述矩阵进行按位分层可以包括：将各通道对应的矩阵中的各十进制元素转化为二进制元素，并根据二进制位对各所述通道对应的矩阵进行按位分层。
本示例性实施例中，每个通道对应的矩阵中的每个元素都是十进制元素，可以将所有十进制元素均转换为对应的二进制元素，具体地转换过程可以通过程序而执行。每个通道可以对应多个不同的矩阵，且每个矩阵中的每个元素可以对应一个像素。本示例中可以对各通道对应的所有矩阵中的元素进行转换，从而进行按位分层。例如，可以以R通道的部分矩阵为例说明按位分层的具体步骤。
按位分层可以理解为按照二进制元素的位数，对矩阵中的各二进制元素进行分层，例如，二进制元素位数为8，则可以将每个二进制元素对应的分解为8个矩阵。
例如，R通道的一小部分矩阵如公式(1)所示：
Figure PCTCN2018097010-appb-000001
将公式(1)对应的矩阵中的各个元素转换的对应的二进制矩阵如公式(2)所示：
Figure PCTCN2018097010-appb-000002
R通道矩阵的每个元素都可以按二进制位分层，每个通道都可以按照二进制位分解为8个矩阵。例如十进制55可以转换为二进制00110111，则元素55可以分解为[0 0 1 1 0 1 1 1]；十进制167可以转换为二进制10100111，则元素167可以对应的分解为[1 0 1 0 0 1 1 1]。R通道对应的部分矩阵可以分解为如公式(3)所示的8个矩阵，由于这8个矩阵所有的元素只包含0和1，因此每个通道对应的矩阵均可以按照二进制位分解为8个01矩阵。
Figure PCTCN2018097010-appb-000003
除此之外，也可以按照上述步骤对G通道和B通道对应的矩阵，根据二进制位进行按位分层，从而得到与G通道对应的8个01矩阵G1、G2、G3、G4、G5、G6、G7、G8；以及与B通道对应的8个01矩阵B1、B2、B3、B4、B5、B6、B7、B8。通过本示例中的按位分层，可以避免图片传输过程中大量冗余信息的影响，从而减少图片的存储量。
进一步地，本示例实施方式中，对各所述矩阵进行按位分层后，所述方法还可以包括：对各所述通道每一层对应的矩阵进行相关性计算，并获取与所述相关性对应的差异矩阵；对所述差异矩阵进行行程长度编码，获取与各所述通道每一层对应的多个所述图片位层数据。
本示例性实施例中，首先可以对分解的三个通道的图片进行灰度化处理，如图2所示。从图2中可以发现，各通道对应的灰度图具有相似性，因此可以预测分解的三组矩阵R1-8、G1-8和B1-8的对应层之间必然存在强相关性。为了验证三组分解矩阵之间的相关性，本示例中对各所述通道的每一层对应的矩阵进行相关性计算。具体步骤如下：
具体而言，三组分解矩阵之间的对应关系如图3所示，由所述对应关系图可以看出，三个通道的相同层的分解矩阵之间一一对应，例如R1分别与G1和B1对应；R8分别与G8以及B8对应。
在得出三个通道的相同层的分解矩阵之间相关性之后，可以用差异矩阵表示该相关性，所述差异矩阵可以使用异或操作完成。进一步地，G通道和B通道的每一层的分解矩阵都可以用R通道与该层对应的分解矩阵结合差异矩阵来表示。由图4可以看出，三 个通道在第8层的分解矩阵对应的二值图片之间也具有较强的相关性，例如R8对应的二值图像可以看作是由G8对应的二值图像和B8对应的二值图像组合而成。除此之外，三个通道第一层的分解矩阵R1、G1和B1可以如公式(4)所示。
Figure PCTCN2018097010-appb-000004
由公式(4)可以看出，G1与R1之间只有一个元素不相同，B1与G1之间有两个元素不相同。因此，G1和B1均可以通过R1加上差异矩阵的形式进行表示，例如公式(5)所示，且差异矩阵可以通过异或操作完成。由此可以看出差异矩阵G1'和B1'都是稀疏矩阵，此时可以记录R1，G1'和B1'。需要注意的是，进行相关性计算并不要求必须使用R通道，也就是说，除此之外，也可以通过G通道与差异矩阵R'和B'描述三个通道的相同层的分解矩阵之间的相关性；还可以通过B通道与差异矩阵R'和G'描述三个通道的相同层的分解矩阵之间的相关性。
Figure PCTCN2018097010-appb-000005
接下来可以对所述差异矩阵进行行程长度编码(RLE，Run Length Encoding)，获取与各所述通道每一层对应的多个图片位层数据。所述行程长度编码方式是一种无损压缩方式，其可以将一扫描行中的颜色值相同的相邻像素用一个计数值和那些像素的颜色值来代替。例如:aaabccccccddeee，则可用3a1b6c2d3e来代替。同样的，字符串JJJJJJAAAAVVVVAAAAAA可以被编码为6J4A4V6A的形式，其中"6J"意味着6个字符J，"4A"意味着4个字符A，通过行程长度编码方式可以避免大量的冗余信息，使图片处 理过程更加简单，速度更快。
本示例中，可以对所述差异矩阵进行行程长度编码，以得到与各所述通道每一层对应的多个图片位层数据，且图片位层数据与矩阵一一对应。图片位层数据可以为数据序列，例如，可以对各稀疏矩阵进行行程长度编码，得到24个数据序列且可以通过这24个数据序列完全表达所述图片。例如，矩阵B1'经过压缩编码后得到的数据序列为01410。所述24各数据序列用可以表示为：
Figure PCTCN2018097010-appb-000006
在步骤S120中，根据一倒排索引结构存储按位分层后的各所述通道对应的图片位层数据，并生成一图层索引记录文件。
本示例性实施例中，可以首先建立一个倒排索引结构，所述倒排索引结构可以在实际应用中根据属性值来确定记录的位置。所述倒排索引结构可以为一索引表，所述索引表的第一列可以用于存储行程长度编码得到的图片位层数据；第二列可以用于存储根据图片位层数据计算的索引值，该索引值可以包括HashCode索引值或MD5索引值；所述索引表的第三列可以用于存储图层索引。如果图片为A、B、C、D，则倒排索引结构可以如表1所示。所述图层索引记录文件可以用于记录根据所述倒排索引结构存储所有图片位层数据的过程。
位层数据 索引值 图层索引
序列1 486dfg4dfg4s3245 图A-R1，图D-R3
序列2 Dfg765gdfs76dgf7 图B-R1，图C-R7
序列3 gfGREdsfsdf89fds 图A-R2
序列4 H09jfjfdni0923h4 图C-B3
…… …… ……
表1
具体地，本示例实施方式中，根据一倒排索引结构存储按位分层后的各所述通道对应的图片位层数据可以包括：计算所述图片位层数据的索引值，并在所述倒排索引结构中查找与所述索引值匹配的索引行；在未查找到与所述索引值匹配的所述索引行时，在所述倒排索引结构中增加一行图层索引记录；在查找到与所述索引值匹配的所述索引行时，检测所述图片位层数据是否完全匹配。
本示例性实施例中，可以首先计算各个图片位层数据对应的索引值，即各通道各层对应的数据序列中存储的数据值，该索引值可以包括HashCode索引值或MD5索引值，例如可以通过函数循环计算和获取各通道各层对应的数据序列的索引值。具体地，可以根据上述步骤对图片进行按位分层，获取24个图片位层数据。以分解图片A为例进行 说明：
Figure PCTCN2018097010-appb-000007
接下来可以在上述索引表中查找与计算的索引值对应或者匹配的索引行。如果未查找到与所述索引值匹配的所述索引行，可以在所述倒排索引结构中增加一行图层索引记录。参考表1进行说明，例如当前计算的图A-R3的索引值为486dgf4dfg4s3246，与表1中的所有索引行均不匹配，则可以将该数据序列，索引值，以及对应的图层索引添加至索引表中。在查找到与所述索引值匹配的所述索引行时，可以检测所述图片位层数据是否完全匹配。依然参考表1进行说明，例如当前计算的图片A第二层R通道的索引值为gfGREdsfsdf89fds，可以继续检测用于计算的数据序列与索引表中存储的数据序列是否完全匹配，此时，可以通过判断数据序列对应的矩阵中的每个元素而判断。
进一步地，本示例实施方式中，检测所述图片位层数据是否完全匹配可以包括：在检测所述图片位层数据完全匹配时，将图层对应的标记添加至所述倒排索引结构中；在检测所述图片位层数据不完全匹配时，重新计算所述索引值。
本示例性实施例中，在检测到当前计算的所述图片位层数据与索引表中存储的图片位层数据不完全匹配或者是完全不匹配时，为了提高查找的准确率，可以重新计算所述索引值；在检测到当前计算的所述图片位层数据与索引表中存储的图片位层数据完全匹配时，可以将该索引值对应的图片的通道和层数标记在索引表中。例如，计算的图片A第二层R通道的索引值为gfGREdsfsdf89fds，且用于计算的数据序列对应的矩阵中的每个元素均与索引表中存储的数据序列的每个元素相同，因此可以将图片A第二层R通道对应的标记图A-R2记录在索引表的图层索引列。
需要补充的是，上述所有的查询与索引过程均可以被记录在图层索引记录文件中，图层索引记录文件中的所有记录可以用于表示完整的图片。
在步骤S130中，根据所述图层索引记录文件对图片的所有所述图层索引进行解码，并根据解码结果生成所述图片。
本示例性实施例中，可以对步骤S120中生成的图层索引记录文件进行解码，根据解码结果生成的所述图片即为被按位分层的图片。解码过程为编码过程的逆过程，因此，编码方法与解码方法可以相互对应。本示例中，可以对某一图片对应的索引表中存储的所有图层索引进行解码，得到最初的与该图片三个通道对应的图片数据，进而得到原始图片。
具体而言，本示例实施方式中，对图片的所有所述图层索引进行解码可以包括：根据所述图层索引记录文件解码图片的所有所述图层索引，并根据所述图层索引查找压缩后的与所述图片对应的所述图片位层数据；对压缩后的所述图片位层数据进行行程长度解码，获取未压缩的与所述图片对应的原始图片位层数据矩阵；对所述原始图片位层数据矩阵进行异或运算，获取未分层的与所述图片的多个通道对应的原矩阵并根据所述原 矩阵合成所述图片。
本示例性实施例中，可以从最底层的数据着手，一层一层地获取可以表示图片的原始数据。首先可以根据所述图层索引记录文件解码图片的所有图层索引，例如获取与图A对应的所有图层索引包括：图A-R1-8，图A-G1-8，图A-B1-8；接下来根据所述图层索引查找压缩后的与所述图层对应的所述图片位层数据，即在索引表中查找与图片A的所有图层索引对应的数据序列，例如得到数据序列01410，需要注意的是，此处获取的数据序列为经过行程长度编码压缩过的数据序列。
进一步地，可以对获取的压缩过的数据序列进行行程长度解码，以获取未经编码压缩处理的原始数据矩阵。例如，可以对数据序列01410进行行程长度解码，得到对应的原始矩阵B1'，此处可以对得到的24个数据序列循环进行解码，以得到未压缩的24个原始图片位层数据矩阵：
Figure PCTCN2018097010-appb-000008
由于上述的得到的24个原始图片位层数据矩阵中的差异矩阵G1'和B1'均可以通过异或操作得到，因此可以对差异矩阵G1'和B1'再次进行异或操作，使其恢复未经过任何处理的原始矩阵
Figure PCTCN2018097010-appb-000009
此处
Figure PCTCN2018097010-appb-000010
分别表示R、G、B三个通道第一层到第八层对应的矩阵，可以分别用于表示RGB三个通道，最后可以通过三个通道合成最初的原始图片。通过将图片信息编码压缩，可以节约网络传输带宽，从而提高图片传输速率，再通过将图片信息进行解码，可以节省存储空间。
此外，本示例实施方式中，所述方法还可以包括：通过所述索引值对根据解码结果生成的所述图片进行校验。
本示例性实施例中，为了保证图片传输的准确性，可以对解码后生成的图片进行校验，判断该图片与原始的被按位分层的图片是否一致；当然也可以不进行校验，具体的校验过程可以通过程序完成，此处不再赘述。
需要补充的是，本示例中具体描述的只是对一张图片的处理过程，进一步地可以通过循环结合本示例中的按位分层、编码存储以及解码等过程，对海量图片进行处理。本示例中，从减少海量图片信息之间的冗余信息入手，对图片进行按层分解并以层的维度进行合并存储，减少了图片存储量和传输量，节约了存储空间，进而提高了传输效率。
本示例实施方式中还提供了一种海量图片处理装置，参考图5所示，所述装置500可以包括：
分层模块501，可以用于获取海量图片中各图片的多个通道对应的矩阵，并对各所述矩阵进行按位分层；
存储模块502，可以用于根据一倒排索引结构存储按位分层后的各所述通道对应的图片位层数据，并生成一图层索引记录文件；其中，所述倒排索引结构包括多个图层索 引；
解码模块503，可以用于根据所述图层索引记录文件对所述图片的所有所述图层索引进行解码，并根据解码结果合成所述图片。
上述海量图片处理装置中各模块的具体细节已经在对应的海量图片处理方法中进行了详细描述，因此此处不再赘述。
应当注意，尽管在上文详细描述中提及了用于动作执行的设备的若干模块或者单元，但是这种划分并非强制性的。实际上，根据本公开的实施方式，上文描述的两个或更多模块或者单元的特征和功能可以在一个模块或者单元中具体化。反之，上文描述的一个模块或者单元的特征和功能可以进一步划分为由多个模块或者单元来具体化。
此外，尽管在附图中以特定顺序描述了本公开中方法的各个步骤，但是，这并非要求或者暗示必须按照该特定顺序来执行这些步骤，或是必须执行全部所示的步骤才能实现期望的结果。附加的或备选的，可以省略某些步骤，将多个步骤合并为一个步骤执行，以及/或者将一个步骤分解为多个步骤执行等。
在本公开的示例性实施例中，还提供了一种能够实现上述方法的电子设备。
所属技术领域的技术人员能够理解，本公开的各个方面可以实现为系统、方法或程序产品。因此，本公开的各个方面可以具体实现为以下形式，即：完全的硬件实施方式、完全的软件实施方式(包括固件、微代码等)，或硬件和软件方面结合的实施方式，这里可以统称为“电路”、“模块”或“系统”。
下面参照图6来描述根据本公开的这种实施方式的电子设备600。图6显示的电子设备600仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图6所示，电子设备600以通用计算设备的形式表现。电子设备600的组件可以包括但不限于：上述至少一个处理单元610、上述至少一个存储单元620、连接不同系统组件(包括存储单元620和处理单元610)的总线630。
其中，所述存储单元存储有程序代码，所述程序代码可以被所述处理单元610执行，使得所述处理单元610执行本说明书上述“示例性方法”部分中描述的根据本公开各种示例性实施方式的步骤。例如，所述处理单元610可以执行上述步骤。
存储单元620可以包括易失性存储单元形式的可读介质，例如随机存取存储单元(RAM)6201和/或高速缓存存储单元6202，还可以进一步包括只读存储单元(ROM)6203。
存储单元620还可以包括具有一组(至少一个)程序模块6205的程序/实用工具6204，这样的程序模块6205包括但不限于：操作系统、一个或者多个应用程序、其它程序模块以及程序数据，这些示例中的每一个或某种组合中可能包括网络环境的实现。
总线630可以为表示几类总线结构中的一种或多种，包括存储单元总线或者存储单元控制器、外围总线、图形加速端口、处理单元或者使用多种总线结构中的任意总线结 构的局域总线。
电子设备600也可以与一个或多个外部设备700(例如键盘、指向设备、蓝牙设备等)通信，还可与一个或者多个使得用户能与该电子设备600交互的设备通信，和/或与使得该电子设备600能与一个或多个其它计算设备进行通信的任何设备(例如路由器、调制解调器等等)通信。这种通信可以通过输入/输出(I/O)接口650进行。并且，电子设备600还可以通过网络适配器660与一个或者多个网络(例如局域网(LAN)，广域网(WAN)和/或公共网络，例如因特网)通信。如图所示，网络适配器660通过总线630与电子设备600的其它模块通信。应当明白，尽管图中未示出，可以结合电子设备600使用其它硬件和/或软件模块，包括但不限于：微代码、设备驱动器、冗余处理单元、外部磁盘驱动阵列、RAID系统、磁带驱动器以及数据备份存储系统等。
通过以上的实施方式的描述，本领域的技术人员易于理解，这里描述的示例实施方式可以通过软件实现，也可以通过软件结合必要的硬件的方式来实现。因此，根据本公开实施方式的技术方案可以以软件产品的形式体现出来，该软件产品可以存储在一个非易失性存储介质(可以是CD-ROM，U盘，移动硬盘等)中或网络上，包括若干指令以使得一台计算设备(可以是个人计算机、服务器、终端装置、或者网络设备等)执行根据本公开实施方式的方法。
在本公开的示例性实施例中，还提供了一种计算机可读存储介质，其上存储有能够实现本说明书上述方法的程序产品。在一些可能的实施方式中，本公开的各个方面还可以实现为一种程序产品的形式，其包括程序代码，当所述程序产品在终端设备上运行时，所述程序代码用于使所述终端设备执行本说明书上述“示例性方法”部分中描述的根据本公开各种示例性实施方式的步骤。
参考图7所示，描述了根据本公开的实施方式的用于实现上述方法的程序产品800，其可以采用便携式紧凑盘只读存储器(CD-ROM)并包括程序代码，并可以在终端设备，例如个人电脑上运行。然而，本公开的程序产品不限于此，在本文件中，可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。
所述程序产品可以采用一个或多个可读介质的任意组合。可读介质可以是可读信号介质或者可读存储介质。可读存储介质例如可以为但不限于电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。可读存储介质的更具体的例子(非穷举的列表)包括：具有一个或多个导线的电连接、便携式盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。
计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中 承载了可读程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。可读信号介质还可以是可读存储介质以外的任何可读介质，该可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。
可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于无线、有线、光缆、RF等等，或者上述的任意合适的组合。
可以以一种或多种程序设计语言的任意组合来编写用于执行本公开操作的程序代码，所述程序设计语言包括面向对象的程序设计语言—诸如Java、C++等，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算设备上执行、部分地在用户设备上执行、作为一个独立的软件包执行、部分在用户计算设备上部分在远程计算设备上执行、或者完全在远程计算设备或服务器上执行。在涉及远程计算设备的情形中，远程计算设备可以通过任意种类的网络，包括局域网(LAN)或广域网(WAN)，连接到用户计算设备，或者，可以连接到外部计算设备(例如利用因特网服务提供商来通过因特网连接)。
此外，上述附图仅是根据本公开示例性实施例的方法所包括的处理的示意性说明，而不是限制目的。易于理解，上述附图所示的处理并不表明或限制这些处理的时间顺序。另外，也易于理解，这些处理可以是例如在多个模块中同步或异步执行的。
本领域技术人员在考虑说明书及实践这里公开的发明后，将容易想到本公开的其他实施例。本公开旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性的，本公开的真正范围和精神由权利要求指出。

Claims (10)

  1. 一种海量图片处理方法，其特征在于，包括：
    获取海量图片中各图片的多个通道对应的矩阵，并对各所述矩阵进行按位分层；
    根据一倒排索引结构存储按位分层后的各所述通道对应的图片位层数据，并生成一图层索引记录文件；其中，所述倒排索引结构包括多个图层索引；
    根据所述图层索引记录文件对所述图片的所有所述图层索引进行解码，并根据解码结果合成所述图片。
  2. 根据权利要求1所述的海量图片处理方法，其特征在于，对各所述矩阵进行按位分层包括：
    将各通道对应的矩阵中的各十进制元素转化为二进制元素，并根据二进制位对各所述通道对应的矩阵进行按位分层。
  3. 根据权利要求1所述的海量图片处理方法，其特征在于，对各所述矩阵进行按位分层后，所述方法还包括：
    对各所述通道每一层对应的矩阵进行相关性计算，并获取与所述相关性对应的差异矩阵；
    对所述差异矩阵进行行程长度编码，获取与各所述通道每一层对应的多个所述图片位层数据。
  4. 根据权利要求1所述的海量图片处理方法，其特征在于，根据一倒排索引结构存储按位分层后的各所述通道对应的图片位层数据包括：
    计算所述图片位层数据的索引值，并在所述倒排索引结构中查找与所述索引值匹配的索引行；
    在未查找到与所述索引值匹配的所述索引行时，在所述倒排索引结构中增加一行图层索引记录；
    在查找到与所述索引值匹配的所述索引行时，检测所述图片位层数据是否完全匹配。
  5. 根据权利要求4所述的海量图片处理方法，其特征在于，检测所述图片位层数据是否完全匹配包括：
    在检测所述图片位层数据完全匹配时，将图层对应的标记添加至所述倒排索引结构中；
    在检测所述图片位层数据不完全匹配时，重新计算所述索引值。
  6. 根据权利要求1所述的海量图片处理方法，其特征在于，对所述图片的所有所述图层索引进行解码包括：
    根据所述图层索引记录文件解码所述图片的所有所述图层索引，并根据所述图层索引查找压缩后的与所述图片对应的所述图片位层数据；
    对压缩后的所述图片位层数据进行行程长度解码，获取未压缩的与所述图片对应的原始图片位层数据矩阵；
    对所述原始图片位层数据矩阵进行异或运算，获取未分层的与所述图片的多个通道对应的原矩阵并根据所述原矩阵合成所述图片。
  7. 根据权利要求4所述的海量图片处理方法，其特征在于，所述方法还包括：
    通过所述索引值对根据解码结果生成的所述图片进行校验。
  8. 一种海量图片处理装置，其特征在于，包括：
    分层模块，用于获取海量图片中各图片的多个通道对应的矩阵，并对各所述矩阵进行按位分层；
    存储模块，用于根据一倒排索引结构存储按位分层后的各所述通道对应的图片位层数据，并生成一图层索引记录文件；其中，所述倒排索引结构包括多个图层索引；
    解码模块，用于根据所述图层索引记录文件对所述图片的所有所述图层索引进行解码，并根据解码结果合成所述图片。
  9. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1-7任一项所述的海量图片处理方法。
  10. 一种电子设备，其特征在于，包括：
    处理器；以及
    存储器，用于存储所述处理器的可执行指令；
    其中，所述处理器配置为经由执行所述可执行指令来执行权利要求1-7任一项所述的海量图片处理方法。
PCT/CN2018/097010 2017-09-08 2018-07-25 海量图片处理方法、装置、电子设备及存储介质 Ceased WO2019047628A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/642,095 US11395010B2 (en) 2017-09-08 2018-07-25 Massive picture processing method converting decimal element in matrices into binary element

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710804971.5 2017-09-08
CN201710804971.5A CN110019865B (zh) 2017-09-08 2017-09-08 海量图片处理方法、装置、电子设备及存储介质

Publications (1)

Publication Number Publication Date
WO2019047628A1 true WO2019047628A1 (zh) 2019-03-14

Family

ID=65633492

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/097010 Ceased WO2019047628A1 (zh) 2017-09-08 2018-07-25 海量图片处理方法、装置、电子设备及存储介质

Country Status (3)

Country Link
US (1) US11395010B2 (zh)
CN (1) CN110019865B (zh)
WO (1) WO2019047628A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116301655A (zh) * 2023-04-12 2023-06-23 深圳市英唐数码科技有限公司 一种历史笔记图片加载方法、系统和可读存储介质
CN117097906A (zh) * 2023-10-20 2023-11-21 河北天英软件科技有限公司 一种区域医疗资源高效利用的方法及系统
CN119559295A (zh) * 2025-01-26 2025-03-04 山东云海国创云计算装备产业创新中心有限公司 二值图孔洞填充方法、装置、设备及可读存储介质

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111368023A (zh) * 2020-03-29 2020-07-03 北京量子时代智能设备有限公司 一种安全的矩阵式文本数据存储表达方法
CN113627320B (zh) * 2021-08-09 2023-09-26 东云睿连(武汉)计算技术有限公司 一种基于计算机视觉的工程图纸比对装置和方法
CN116016606B (zh) * 2023-01-09 2023-08-18 安徽省分众分析测试技术有限公司 一种基于智慧云的污水处理运维数据高效管理系统
CN116541828B (zh) * 2023-07-03 2023-09-22 北京双鑫汇在线科技有限公司 一种服务信息数据的智能管理方法
CN116993611B (zh) * 2023-07-28 2025-02-28 重庆赛力斯凤凰智创科技有限公司 图像加载方法、装置、电子设备及可读存储介质
CN117201603B (zh) * 2023-11-06 2024-02-06 苔花科迈(西安)信息技术有限公司 基于二维码的煤矿井下数据交接方法及系统

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101030365A (zh) * 2007-04-10 2007-09-05 北京中星微电子有限公司 一种数字图像的存储、显示方法及装置
CN101930614A (zh) * 2010-08-10 2010-12-29 西安交通大学 基于视频分层的绘画渲染方法
US20120243784A1 (en) * 2011-03-24 2012-09-27 Takahiro Fukuhara Image processing device and method

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9251197B2 (en) 2011-06-27 2016-02-02 Jethrodata Ltd. System, method and data structure for fast loading, storing and access to huge data sets in real time
US20140092971A1 (en) * 2012-09-28 2014-04-03 Kiran Mukesh Misra Picture processing in scalable video systems
CN103731448B (zh) 2012-10-11 2017-09-26 腾讯科技（深圳）有限公司 一种压缩图片的方法和系统
CN103955514A (zh) * 2014-05-05 2014-07-30 陈浩 一种基于Lucene倒排索引的图像特征索引方法
CN105245888A (zh) * 2014-06-11 2016-01-13 富士通株式会社 图像压缩方法和图像压缩装置
GB2531271A (en) * 2014-10-14 2016-04-20 Nokia Technologies Oy An apparatus, a method and a computer program for image sequence coding and decoding
CN106612435A (zh) * 2016-01-16 2017-05-03 四川用联信息技术有限公司 一种基于svd-dwt-dct的联合图像压缩方法
CN106023122A (zh) * 2016-05-05 2016-10-12 江南大学 基于多通道分解的图像融合方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101030365A (zh) * 2007-04-10 2007-09-05 北京中星微电子有限公司 一种数字图像的存储、显示方法及装置
CN101930614A (zh) * 2010-08-10 2010-12-29 西安交通大学 基于视频分层的绘画渲染方法
US20120243784A1 (en) * 2011-03-24 2012-09-27 Takahiro Fukuhara Image processing device and method

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116301655A (zh) * 2023-04-12 2023-06-23 深圳市英唐数码科技有限公司 一种历史笔记图片加载方法、系统和可读存储介质
CN116301655B (zh) * 2023-04-12 2023-11-07 深圳市英唐数码科技有限公司 一种历史笔记图片加载方法、系统和可读存储介质
CN117097906A (zh) * 2023-10-20 2023-11-21 河北天英软件科技有限公司 一种区域医疗资源高效利用的方法及系统
CN117097906B (zh) * 2023-10-20 2023-12-26 河北天英软件科技有限公司 一种区域医疗资源高效利用的方法及系统
CN119559295A (zh) * 2025-01-26 2025-03-04 山东云海国创云计算装备产业创新中心有限公司 二值图孔洞填充方法、装置、设备及可读存储介质

Also Published As

Publication number Publication date
US11395010B2 (en) 2022-07-19
CN110019865A (zh) 2019-07-16
US20200204832A1 (en) 2020-06-25
CN110019865B (zh) 2021-01-26

Similar Documents

Publication Publication Date Title
WO2019047628A1 (zh) 海量图片处理方法、装置、电子设备及存储介质
CN110879854B (zh) 使用超集树数据结构搜索数据
US11030149B2 (en) File format for accessing data quickly and efficiently
WO2019153867A1 (zh) 二维码生成及识别
EP3791331A1 (en) Efficient data encoding for deep neural network training
CN113327599B (zh) 语音识别方法、装置、介质及电子设备
US20140232574A1 (en) System, method and non-transitory computer readable medium for compressing genetic information
ES2898868T3 (es) Método y dispositivo para la búsqueda de imágenes
WO2020119487A1 (zh) 点云的编解码方法及装置
US9544596B1 (en) Optimized template matching approach to intra-coding in video/image compression
US20250175192A1 (en) System and method for secure data processing with privacy-preserving compression and quality enhancement
US20240421830A1 (en) Unified platform for multi-type data compression and decompression using homomorphic encryption and neural upsampling
US20240314293A1 (en) Lossless image compression using block based prediction and optimized context adaptive entropy coding
CN109474826B (zh) 图片压缩方法、装置、电子设备及存储介质
WO2020192012A1 (zh) 一种数据处理方法、设备及存储介质
CN115643348B (zh) 基于可逆图像处理网络的可证安全自然隐写方法及装置
US11138151B2 (en) Compression scheme for floating point values
CN115802060A (zh) 语义通信视频传输方法及相关设备
CN117939127A (zh) 一种图像处理方法及其相关设备
CN106446272A (zh) 用于发送数据的方法和装置
CN118118164A (zh) 数据保护方法、基于纵向联邦学习模型的预测方法及装置
WO2018196817A1 (zh) 一种基于lz77的数据压缩方法、设备及终端
US9271010B2 (en) System and method for motion estimation for large-size block
CN112486976A (zh) 数据处理方法、装置、网络设备及存储介质
US12192467B1 (en) Arithmetic encoding and decoding method based on semantic source and related device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18853691

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 19.08.2020)

122 Ep: pct application non-entry in european phase

Ref document number: 18853691

Country of ref document: EP

Kind code of ref document: A1

WWG Wipo information: grant in national office

Ref document number: 202037006328

Country of ref document: IN