CN105677763B - A kind of image quality measure system based on Hadoop - Google Patents
A kind of image quality measure system based on Hadoop Download PDFInfo
- Publication number
- CN105677763B CN105677763B CN201511022591.3A CN201511022591A CN105677763B CN 105677763 B CN105677763 B CN 105677763B CN 201511022591 A CN201511022591 A CN 201511022591A CN 105677763 B CN105677763 B CN 105677763B
- Authority
- CN
- China
- Prior art keywords
- image
- image quality
- hadoop
- mapreduce
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
Abstract
The image quality measure system based on Hadoop that the invention discloses a kind of, including client and Hadoop cluster；Client include service selecting module and with picture transfer module；Hadoop cluster includes host node and multiple child nodes；The host node is responsible for the coordination execution of the initialization of operation, the distribution of operation, operation, while being responsible for the file system of management cluster；The child node is equipped with image quality evaluation module, is responsible for the execution and data block storage of map task and reduce task；The host node is equipped with the communication server, and the communication server is responsible for receiving the image sended over from client and opening a MapReduce operation for each user carrying out image quality measure.The present invention utilizes the Distributed Parallel Computing advantage of Hadoop cluster, effectively shortens processing time of a large amount of picture quality evaluations, and the user experience is improved.
Description
Technical field
The present invention relates to image intelligent process field, in particular to a kind of image quality measure system based on Hadoop.
Background technique
In recent years, image quality measure because its in various application potential great demand, cause many scholars
Concern.Image quality measure can help people to pick out high artistic image, filter out low artistic image, people from irksome
Image management work in free.For example, in image retrieval, it is intended that computer can not only be according to picture material
Image can be retrieved according to picture quality.
Nowadays most of scholar is intended to comment using a large amount of image study picture quality about the work of image quality measure
Estimate model, classification and the forecast image mass fraction of height aesthetic feeling are carried out to image.In order to improve the accuracy of system, Ren Menqiang
Adjust the importance of image characteristics extraction, including manual feature and local feature.In order to further increase accuracy, huge figure is established
As database, comment, geography information of user etc. are also considered by system.Generally speaking, existing system lacks to picture quality
Assess the attention of efficiency and user experience.
With popularizing for smart phone, people are obtained and storage image is increasingly easy, and amount of images is also increasing severely.People
Expected image quality evaluation algorithm can help the image on managing mobile phone.Since image quality measure process is complicated, time-consuming,
Operation image quality evaluation algorithm process speed is slow on mobile phone, inefficiency, especially when amount of images is huge.
Hadoop is made of numerous computer nodes, is realized distributed storage and parallel computation, is suitble to the big number of processing
According to open source software basic framework, be widely used in cloud computing.The appearance of cloud computing technology provides newly for image procossing
Thinking.By combining cloud computing technology, image processing efficiency is greatly improved.University Of Chongqing Zhang little Hong et al. invents one kind and is based on
The search method (application number: 201310038448.8, publication number: 103207889A) of the magnanimity facial image of Hadoop.This is specially
It is sharp mainly to realize image retrieval function using Hadoop, including image data is established and is indexed, and carries out distributed search completion
The fast search of facial image.Due to lacking the design of client, the face identification method of the patent of invention may not apply to move
In dynamic terminal.
By cloud computing resource abundant, Hadoop can mitigate the heavy computation burden of terminal significantly.However, at present
It is not found also both at home and abroad Hadoop technical application on image quality measure.In addition, Hadoop is initially designed to big text
The processing of data needs to do Curve guide impeller again for the processing of image data.
Summary of the invention
In order to overcome the disadvantages mentioned above and deficiency of the prior art, the purpose of the present invention is to provide a kind of based on Hadoop's
Image quality measure system, image quality measure algorithm complicated, time-consuming in client can be moved to has abundant calculating money
The cloud in source, by flow exchange for image quality measure high efficiency and good user experience.
The purpose of the present invention is achieved through the following technical solutions:
A kind of image quality measure system based on Hadoop, including client and Hadoop cluster；
The client includes service selecting module and picture transfer module；The service selecting module is for selecting user
Image and the service of quality evaluation are needed, and shows the result of the image quality measure returned from server end；
The picture transfer module is used to send user's request, transmission user images to server end by internet, with
And receive the result returned from server end；
The Hadoop cluster includes host node and multiple child nodes；The host node is responsible for the initialization of operation, operation
Distribution, operation coordination execute, while be responsible for management cluster file system；The child node is equipped with image quality measure
Module is responsible for the execution and data block storage of map task and reduce task；The host node is equipped with the communication server,
The communication server is responsible for receiving the image of client transmission and opening a MapReduce operation for each user carrying out figure
As quality evaluation.
The course of work of the communication server is as follows:
Firstly, the image of each user received is stored under particular category, and the catalogue is uploaded into HDFS；So
Afterwards, a MapReduce operation is opened for each user；Finally, MapReduce calls the image matter of image quality evaluation module
Measure the image of the C++ dynamic link library processing input of assessment algorithm.
It is described to open a MapReduce operation progress image quality measure, specifically:
(1) it establishes MapReduce workflow: establishing the workflow of MapReduce processing multi-user's request, establishes
MapReduce handles the workflow of more images；
(2) it defines the input data type of image file: defining the data type ImgFile class of image input, image text
Part input format ImgFileInputFormat class, image key-value pair read in format ImgFileRecordReader class；
(3) image quality measure is realized in map function: realizing MapReduce to the algorithm and reality of image quality measure
Existing MapReduce function.
It is described to establish MapReduce workflow, specifically:
(1-1) establishes the workflow of MapReduce parallel processing multi-user request: the data of each user are as one
The parallel processing of more operations is realized in the input of MapReduce operation by job scheduler；
(1-2) establishes the workflow that MapReduce handles more images: using whole image as an input fragment, and
It is recorded entire fragment as one；In the map stage, every image of user will be counted as a fragment, by realizing image
The map function of quality evaluation algorithm is handled；Input picture by<image file name, picture material>key-value pair form indicate, it is defeated
Result is by<image file name out, image quality measure result>key-value pair form indicate；In the reduce stage, reduce task
The result of map task is summarized and exports a text file.
The input data type for defining image file, specifically:
(2-1) defines the data type ImgFile class of image input, and image input ImgFile class realizes that Writable connects
Mouthful, define getImage () member function, getHeight () member function, getWidth () member function；
(2-2) defines image file input format ImgFileInputFormat class, image file input format
ImgFileInputFormat Similar integral FlieInputFormat class supports ImgFile class, by the image file cutting of input
At input sliced fashion, increase the definition that image file is read, with a width complete image for a fragment, without file point
It cuts；
(2-3) defines image key-value pair and reads in format ImgFileRecordReader class, and image key-value pair reads in format
Input key-value pair is defined as < image file name, in image by ImgFileRecordReader Similar integral RecordReader class
Hold > form, and read from InputSplit the key-value pair of record for Mapper processing；Described image filename is Text class
One example of type, described image content are an examples of ImgFile type.
It is described to realize image quality measure in map function, specifically:
(3-1) realizes that MapReduce to the calling of image quality measure algorithm, i.e., is write with Java language
MapReduce program using JNI interface call by C++ code development image quality measure algorithm, realize image classification and
Prediction of quality, detailed process are as follows:
Firstly, generating the C++ source file of the core algorithm of image quality measure；Then C++ source file being compiled into can add
So file for carrying and calling；Finally, in the data file by trained classification and assessment models storage, so that algorithm is executing
It is called in task；
(3-2) realizes MapReduce function, i.e., map and reduce function is realized in map class and reduce class, specifically
Are as follows:
In the map stage, firstly, it is ImgFile type that map function, which receives an image and reads in, as key-value pair < image
Filename, picture material > in picture material；Then, it obtains the picture material of key-value pair and converts thereof into int array；It connects
, the image that int array is input in JNI function setSourceImageJni () in setting C++ code is inputted；Finally,
Carry out region division, feature extraction and classification assessment processing to it with function Evaluate () or Classify () again；
In the reduce stage, reduce function summarizes the classification or Score on Prediction result of every image of map function output
Into a text file.
The Hadoop cluster achieves tool Hadoop Archives using Hadoop and saves cluster host node namenode
Memory, specifically:
Firstly, being picture archiving with Hadoop Archives before carrying out image quality measure using MapReduce
HAR file；Then, HAR file is uploaded to HDFS；Finally handled using HAR file as the input of MapReduce.
The Hadoop cluster carries out the operating mode reused when image quality measure using task JVM, i.e., each map appoints
An identical JVM may be reused in business, specifically:
In HDOOP_CONF_DIR/mapred-site.xml file
Mapred.job.reuse.jvm.num.tasks parameter is set as -1, Java Virtual Machine and reuses infinitely
It is secondary.
The Hadoop cluster carries out using Fair Scheduler job scheduling strategy when image quality measure, according to each
Size, uplink time schedule job and the reasonable distribution computing resource of a operation, allow all operations that can obtain cluster resource
Most Fairshare.
Communication between the client and Hadoop cluster uses socket technology.
Compared with prior art, the present invention has the following advantages and beneficial effects:
(1) the image quality measure system of the invention based on Hadoop, by complicated image quality measure algorithm
Cloud is moved to, the application in client becomes simple, becomes light client；
(2) the image quality measure system of the invention based on Hadoop, user data beyond the clouds distributed storage, efficiently
Parallel processing, by the design using cloud computing resource abundant and optimization MapReduce, the request of client can be obtained fastly
Speed response, the user experience is improved.
(3) the image quality measure system of the invention based on Hadoop, image quality measure algorithm and model can be
Server end is easily accomplished upgrading, and client does not need to make any change on hardware and software, and application is more convenient.
(4) the image quality measure system of the invention based on Hadoop handles picture construction for Hadoop
MapReduce frame realizes the parallel processing of great amount of images, has good scalability, extends to high-volume image text
The applications such as the conversion of part format, video mode identification.
Detailed description of the invention
Fig. 1 is the image quality measure system physical frame based on Hadoop of the embodiment of the present invention.
Fig. 2 is the image quality measure working-flow figure of the embodiment of the present invention.
Fig. 3 is the work flow diagram of the image quality evaluation module of the embodiment of the present invention.
Fig. 4 is the flow chart of the MapReduce operation of the embodiment of the present invention.
Specific embodiment
Below with reference to embodiment, the present invention is described in further detail, embodiments of the present invention are not limited thereto.
Embodiment
As shown in Figure 1, a kind of image quality measure system based on Hadoop, including client and Hadoop cluster；
Hadoop cluster includes host node and multiple child nodes.Communication service is run in client and Hadoop cluster simultaneously, is responsible for
Data are transmitted, including uploads image from client and returns to image quality measure result；The communication service uses socket skill
Art.
Client include service selecting module and with picture transfer module；The service selecting module is for selecting user to need
Image and the service of quality evaluation are wanted, and shows the result of the image quality measure returned from server end；The picture passes
Defeated module is used to send user's request, transmission user images to server end by internet, and receives and return from server end
The result returned.
As shown in Fig. 2, host node is responsible for the coordination execution of the initialization of operation, the distribution of operation, operation, while being responsible for pipe
Manage the file system of cluster；The child node is equipped with image quality evaluation module, is responsible for holding for map task and reduce task
The storage of capable and data block；The host node is equipped with the communication server, and the communication server is responsible for receiving to be sent out from client
The image brought simultaneously opens a MapReduce operation for each user and carries out image quality measure.
As shown in figure 3, the workflow of the image quality evaluation module of the present embodiment is as follows:
Firstly, extracting aesthetic features to image data base；Then, quality evaluation mould is established by machine training and study
Type, including image aesthetics classifier and image aesthetics regression model；Finally, utilizing established beauty to the image of user's input
Feel grade classifier and aesthstic regression model realizes the classification of image aesthetic-qualitative level height and the prediction of aesthetic score.
The course of work of the communication server of the present embodiment is as follows:
Firstly, the image of each user received is stored under particular category, and the catalogue is uploaded into HDFS；So
Afterwards, a MapReduce operation is opened for each user；Finally, MapReduce calls the image matter of image quality evaluation module
Measure the image of the C++ dynamic link library processing input of assessment algorithm.
The HDFS is Hadoop distributed file system.
In order to realize that parallel processing has the great amount of images of multi-user using Hadoop, building MapReduce frame has come
At image quality measure, including establishing MapReduce workflow, defining the input data type of image file and in map letter
Image quality measure algorithm is realized in number.Specifically:
(1) MapReduce workflow is established, that is, establishes the workflow of MapReduce processing multi-user's request, establish
MapReduce handles the workflow of more images；
(1-1) establishes the workflow of MapReduce parallel processing multi-user request, i.e., the data of each user are as one
The input of a MapReduce operation, the parallel processing of more operations is realized by job scheduler；
(1-2) establishes the workflow that MapReduce handles more images, as shown in Figure 4.Whole image is defeated as one
Enter fragment, and is recorded entire fragment as one.In the map stage, every image of user will be counted as a fragment, by reality
The map function processing of image quality measure algorithm is showed.Input picture by<image file name, picture material>key-value pair form
Indicate, output result by<image file name, image quality measure result>key-value pair form indicate.In the reduce stage,
The result of map task is summarized and exports a text file by reduce task.
(2) the input data type for defining image file defines the data type ImgFile class of image input, image
File input formats ImgFileInputFormat class, image key-value pair read in format ImgFileRecordReader class.Specifically
Are as follows:
(2-1) defines the data type ImgFile class of image input, and image input ImgFile class realizes that Writable connects
Mouthful, it defines getImage (), getHeight (), a series of member functions such as and getWidth ()；
(2-2) defines image file input format ImgFileInputFormat class, image file input format
ImgFileInputFormat Similar integral FlieInputFormat class supports ImgFile class, by the image file cutting of input
At input sliced fashion, the definition of image file reading is increased, with a width complete image for a fragment, without file point
It cuts；
(2-3) defines image key-value pair and reads in format ImgFileRecordReader class, and image key-value pair reads in format
Input key-value pair is defined as < image file name, in image by ImgFileRecordReader Similar integral RecordReader class
Hold > form, and read from InputSplit the key-value pair of record for Mapper processing.Described image filename is Text class
One example of type, described image content are an examples of ImgFile type.
(3) image quality measure algorithm is realized in map function, i.e. realization MapReduce is to image quality measure algorithm
Calling and realize MapReduce function.Specifically:
(3-1) realizes that MapReduce to the calling of image quality evaluation module, i.e., is write with Java language
MapReduce program using JNI interface call by C++ code development image quality measure algorithm, realize image classification and
Score on Prediction.Specifically:
Firstly, generating the C++ source file of the core algorithm of image quality measure；Then C++ source file being compiled into can add
So file for carrying and calling；Finally, in the data file by trained classification and assessment models storage, so that algorithm is executing
It is called in task.
The image quality measure system based on Hadoop uses Hadoop file cache tool Hadoop
So file and trained classification and assessment models data file are deployed to each section in cluster by Distributed Cache
Point in.
(3-2) realizes MapReduce function, i.e., map and reduce function is realized in map class and reduce class, specifically
Are as follows:
In the map stage, firstly, it is ImgFile type that map function, which receives an image and reads in, as key-value pair < image
Filename, picture material > in picture material；Then, it obtains the picture material of key-value pair and converts thereof into int array；It connects
, the image that int array is input in JNI function setSourceImageJni () in setting C++ code is inputted；Finally,
Carry out the processing such as region division, feature extraction and classification assessment to it with function Evaluate () or Classify () again.
In the reduce stage, reduce function summarizes the classification or Score on Prediction result of every image of map function output
Into a text file.
Performance of the Hadoop in terms of handling large amount of small documents be not good enough, handles a large amount of small images to adapt to Hadoop cluster
The application of file improves system performance, optimizes to Hadoop cluster, including saves memory and task JVM reuse.
The saving memory achieves tool Hadoop Archives using Hadoop and saves cluster host node
Namenode memory, specifically:
Firstly, being picture archiving with Hadoop Archives before carrying out image quality measure using MapReduce
HAR file；Then, HAR file is uploaded to HDFS；Finally handled using HAR file as the input of MapReduce.
The task JVM is reused, i.e., an identical JVM may be reused in each map task.Specifically:
In HDOOP_CONF_DIR/mapred-site.xml file
Mapred.job.reuse.jvm.num.tasks parameter, which is set as -1, Java Virtual Machine, may be reused nothing
Limit time.
Hadoop default uses FIFO scheduler, and concurrent multi-user's different images number is requested, and system performance is not high.For
So that the operation of each user is obtained the most Fairshare of cluster resource, improve user experience, the present embodiment uses Fair
Scheduler job scheduling strategy calculates money according to the schedule jobs such as the size of each operation, uplink time and reasonable distribution
Source, specifically:
In HADOOP_CONF_DIR/mapred-site.xml file
Mapred.jobtracker.taskScheduler parameter setting is at org.apache.hadoop.mapred.FairSchedu
ler.In addition, mapred.fairscheduler.sizebasedweight parameter setting at true, after opening the option,
System can be using job size as one of the determinant of distribution computing resource.
In the present embodiment, the present embodiment is in Dell's precision T5610 work station virtual machine with 64G memory
Hadoop cluster is constructed on VMware Workstation 11, cluster is made of 1 host node and 5 child nodes, is constituted each
The basic software of node includes: Ubuntu 10.04LTS operating system, Hadoop0.21.0, JRE 1.6.0_33, OpenCV-
2.2.0.The configuration of single machine is identical with each node of cluster.The present embodiment is respectively 100 to 1000 figures with 10 groups of quantity
As input, the processing time is that operation starts to operation to handle to complete.Single machine and Hadoop cluster different images number input
Manage the comparison such as table 1 of time:
Table 1Hadoop cluster and single machine processing time comparison
| 100 | 200 | 300 | 400 | 500 | 600 | 700 | 800 | 900 | 1000 | |
| Single machine (second) | 10.01 | 23.30 | 34.69 | 43.16 | 52.76 | 61.39 | 70.23 | 79.77 | 90.70 | 100.70 |
| Cluster (second) | 1.83 | 3.62 | 5.15 | 6.30 | 7.77 | 9.10 | 10.43 | 11.93 | 13.30 | 15.03 |
Pass through above table, it can be seen that the time of the processing time ratio Hadoop cluster of single machine is long very much.Work as picture number
Amount is bigger, and the time of single machine processing operation greatly increases, and the advantage of cluster is more obvious.As it can be seen that the application of Hadoop cluster makes
The efficiency of image quality measure system significantly improves.
The present embodiment passes through the request of simulation concurrent user's different images number, to verify the scheduling plan of Fair Scheduler
Slightly.Concurrent user number is 3, respectively user 1, user 2 and user 3, and the picture number inputted is respectively 5,40 and 500.System
Default the scheduling strategy of FIFO, therefore when the different requests sequence of concurrent user will affect the corresponding operation of each user and complete
Between.3 can request that for operation are sequentially 1-2-3,1-3-2,2-1-3,2-3-1,3-1-2 and 3-2-1.The present embodiment calculates
The average handling time of 6 request sequence of each user under FIFO and Fair Scheduler scheduling strategy and processing time
Standard deviation.Comparison under two kinds of scheduling strategies of FIFO and Fair Scheduler is as shown in table 3:
The comparison of table 2Fair Scheduler and FIFO scheduling strategy
By upper table it follows that first, when system process images quantity is smaller, Fair Scheduler is used to dispatch plan
The average used time ratio FIFO slightly spent is few.Fair Scheduler scheduling strategy effectively shortens the waiting time of operation, makes it
The time it takes ratio FIFO is greatly reduced.One short operation will be completed within reasonable time, even if another user
Long working is also in the process of running.Second, it is small using user's used time standard deviation of Fair Scheduler scheduling strategy
In the standard deviation using FIFO scheduling strategy.Obviously, when using FIFO scheduling strategy different job sequence user waiting
Time is quite different, and Fair Scheduler is relatively stable.As it can be seen that the use of Fair Scheduler scheduling strategy can allow
Cluster resource is shared to each user fairness, improves the experience of user.
The above embodiment is a preferred embodiment of the present invention, but embodiments of the present invention are not by the embodiment
Limitation, other any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present invention,
It should be equivalent substitute mode, be included within the scope of the present invention.
Claims (7)
1. a kind of image quality measure system based on Hadoop, which is characterized in that including client and Hadoop cluster；
The client includes service selecting module and picture transfer module；The service selecting module is for selecting user to need
The image of quality evaluation and service, and show the result of the image quality measure returned from server end；The picture transfer mould
Block is used to send user's request, transmission user images to server end by internet, and receives and return from server end
As a result；
The Hadoop cluster includes host node and multiple child nodes；The host node is responsible for point of the initialization of operation, operation
Match, the coordination of operation execution, while being responsible for the file system of management cluster；The child node is equipped with image quality measure mould
Block is responsible for the execution and data block storage of map task and reduce task；The host node is equipped with the communication server, institute
The communication server is stated to be responsible for receiving the image of client transmission and opening a MapReduce operation for each user carrying out image
Quality evaluation；
The course of work of the communication server is as follows:
Firstly, the image of each user received is stored under particular category, and the catalogue is uploaded into HDFS；Then,
A MapReduce operation is opened for each user；Finally, MapReduce calls the picture quality of image quality evaluation module
The image of the C++ dynamic link library processing input of assessment algorithm；
It is described to open a MapReduce operation progress image quality measure, specifically:
(1) it establishes MapReduce workflow: establishing the workflow of MapReduce processing multi-user's request, establishes
MapReduce handles the workflow of more images；
(2) it defines the input data type of image file: defining the data type ImgFile class of image input, image file is defeated
Entry format ImgFileInputFormat class, image key-value pair read in format ImgFileRecordReader class；
(3) image quality measure is realized in map function: realizing MapReduce to the algorithm and reality of image quality evaluation module
Existing MapReduce function；
It is described to establish MapReduce workflow, specifically:
(1-1) establishes the workflow of MapReduce parallel processing multi-user request: the data of each user are as one
The parallel processing of more operations is realized in the input of MapReduce operation by job scheduler；
(1-2) establishes the workflow that MapReduce handles more images: using whole image as an input fragment, and will be whole
A fragment is recorded as one；In the map stage, every image of user will be counted as a fragment, by realizing picture quality
The map function of assessment algorithm is handled；Input picture by<image file name, picture material>key-value pair form indicate, output knot
Fruit by<image file name, image quality measure result>key-value pair form indicate；In the reduce stage, reduce task will
The result of map task summarizes and exports a text file.
2. the image quality measure system according to claim 1 based on Hadoop, which is characterized in that the definition image
The input data type of file, specifically:
(2-1) defines the data type ImgFile class of image input, and image inputs ImgFile class and realizes Writable interface, fixed
Adopted getImage () member function, getHeight () member function, getWidth () member function；
(2-2) defines image file input format ImgFileInputFormat class, image file input format
ImgFileInputFormat Similar integral FlieInputFormat class supports ImgFile class, by the image file cutting of input
At input sliced fashion, increase the definition that image file is read, with a width complete image for a fragment, without file point
It cuts；
(2-3) defines image key-value pair and reads in format ImgFileRecordReader class, and image key-value pair reads in format
Input key-value pair is defined as < image file name, in image by ImgFileRecordReader Similar integral RecordReader class
Hold > form, and read from InputSplit the key-value pair of record for Mapper processing；Described image filename is Text class
One example of type, described image content are an examples of ImgFile type.
3. the image quality measure system according to claim 1 based on Hadoop, which is characterized in that described in map letter
Image quality measure is realized in number, specifically:
(3-1) realizes calling of the MapReduce to image quality measure algorithm, i.e., the MapReduce journey write with Java language
Sequence calls the image quality measure algorithm by C++ code development using JNI interface, realizes the classification and prediction of quality of image, has
Body process are as follows:
Firstly, generating the C++ source file of the core algorithm of image quality measure；Then C++ source file is compiled into can load and
So file of calling；Finally, in the data file by trained classification and assessment models storage, so that algorithm is in the task of execution
Middle calling；
(3-2) realizes MapReduce function, i.e., map and reduce function is realized in map class and reduce class, specifically:
In the map stage, firstly, it is ImgFile type that map function, which receives an image and reads in, as key-value pair < image file
Name, picture material > in picture material；Then, it obtains the picture material of key-value pair and converts thereof into int array；Then, will
Int array is input to the image input in JNI function setSourceImageJni () in setting C++ code；Finally, again with letter
Number Evaluate () or Classify () carry out region division, feature extraction and classification assessment processing to it；
In the reduce stage, classification that reduce function summarizes every image of map function output or Score on Prediction result are to one
In a text file.
4. the image quality measure system according to claim 1 based on Hadoop, which is characterized in that the Hadoop collection
Group achieves tool Hadoop Archives using Hadoop and saves cluster host node namenode memory, specifically:
Firstly, with Hadoop Archives picture archiving being HAR before carrying out image quality measure using MapReduce
File；Then, HAR file is uploaded to HDFS；Finally handled using HAR file as the input of MapReduce.
5. the image quality measure system according to claim 1 based on Hadoop, which is characterized in that the Hadoop collection
Group carries out the operating mode reused when image quality measure using task JVM, i.e., each map task can reuse a phase
Same JVM, specifically:
The mapred.job.reuse.jvm.num.tasks ginseng in HDOOP_CONF_DIR/mapred-site.xml file
Number is set as -1, Java Virtual Machine and reuses infinitely.
6. the image quality measure system according to claim 1 based on Hadoop, which is characterized in that the Hadoop collection
Group carries out using Fair Scheduler job scheduling strategy when image quality measure, when according to the size of each operation, upload
Between schedule job and reasonable distribution computing resource, allow all operations that can obtain the most Fairshare of cluster resource.
7. the image quality measure system according to claim 1 based on Hadoop, which is characterized in that the client and
Communication between Hadoop cluster uses socket technology.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511022591.3A CN105677763B (en) | 2015-12-29 | 2015-12-29 | A kind of image quality measure system based on Hadoop |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511022591.3A CN105677763B (en) | 2015-12-29 | 2015-12-29 | A kind of image quality measure system based on Hadoop |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105677763A CN105677763A (en) | 2016-06-15 |
| CN105677763B true CN105677763B (en) | 2019-08-20 |
Family
ID=56298092
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201511022591.3A Active CN105677763B (en) | 2015-12-29 | 2015-12-29 | A kind of image quality measure system based on Hadoop |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105677763B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP4141811A1 (en) * | 2021-08-30 | 2023-03-01 | Acer Incorporated | Image processing method |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126601A (en) * | 2016-06-20 | 2016-11-16 | 华南理工大学 | A kind of social security distributed preprocess method of big data and system |
| CN106371908A (en) * | 2016-08-31 | 2017-02-01 | 武汉鸿瑞达信息技术有限公司 | Optimization method for image/video filter task distribution based on PSO (Particle Swarm Optimization) |
| CN106549949A (en) * | 2016-10-31 | 2017-03-29 | 广西东方道迩科技有限公司 | A kind of image data processing system and its image processing method |
| CN108256118B (en) * | 2018-02-13 | 2023-09-22 | 腾讯科技（深圳）有限公司 | Data processing method, device, system, computing equipment and storage medium |
| CN108900335A (en) * | 2018-06-28 | 2018-11-27 | 泰康保险集团股份有限公司 | Workflow management method and device based on Hadoop |
| CN110378332A (en) * | 2019-06-14 | 2019-10-25 | 上海咪啰信息科技有限公司 | A kind of container terminal case number (CN) and Train number recognition method and system |
| CN113836130A (en) * | 2021-09-28 | 2021-12-24 | 深圳创维智慧科技有限公司 | Data quality evaluation method, device, equipment and storage medium |
| CN115422126B (en) * | 2022-11-04 | 2023-03-24 | 浪潮软件股份有限公司 | Method, system and device for rapidly transferring certificate OFD format file to picture |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103207889A (en) * | 2013-01-31 | 2013-07-17 | 重庆大学 | Method for retrieving massive face images based on Hadoop |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9152870B2 (en) * | 2013-03-15 | 2015-10-06 | Sri International | Computer vision as a service |
-
2015
- 2015-12-29 CN CN201511022591.3A patent/CN105677763B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103207889A (en) * | 2013-01-31 | 2013-07-17 | 重庆大学 | Method for retrieving massive face images based on Hadoop |
Non-Patent Citations (2)
| Title |
|---|
| A System of Image Aesthetic Classification and Evaluation Using Cloud Computing;Weining Wang 等;《CCPR 2014》;20141231;第183–195页 |
| 基于Hadoop的外观专利图像检索系统的研究与实现;王贤伟;《中国优秀硕士学位论文全文数据库 信息科技辑》;20111115(第11期);第8-56页 |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP4141811A1 (en) * | 2021-08-30 | 2023-03-01 | Acer Incorporated | Image processing method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105677763A (en) | 2016-06-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105677763B (en) | A kind of image quality measure system based on Hadoop | |
| CN106250987B (en) | A kind of machine learning method, device and big data platform | |
| CN103246749B (en) | The matrix database system and its querying method that Based on Distributed calculates | |
| CN111400326B (en) | Smart city data management system and method thereof | |
| CN109887098B (en) | Web AR data presentation mode based on distributed computing | |
| WO2021143264A1 (en) | Image processing method and apparatus, server and storage medium | |
| CN110968579B (en) | Execution plan generation and execution method, database engine and storage medium | |
| CN108363713A (en) | Video image information resolver, system and method | |
| CN110399446A (en) | Method for visualizing, device, equipment and the storage medium of extensive space-time data | |
| WO2016165321A1 (en) | Method and apparatus for establishing requirement meta model for high-speed train | |
| CN112365355B (en) | Method, device and readable medium for calculating foundation valuation and risk index in real time | |
| CN103699656A (en) | GPU-based mass-multimedia-data-oriented MapReduce platform | |
| CN110059309A (en) | The generation method and device of information object title | |
| Shahrbanoo et al. | An Approach for agile SOA development using agile principals | |
| CN103198099A (en) | Cloud-based data mining application method facing telecommunication service | |
| CN105357546A (en) | Cloud computing based video transcoding method | |
| CN110413776A (en) | It is a kind of to cooperate with parallel text subject model LDA high-performance calculation method based on CPU-GPU | |
| CN110489749B (en) | Business process optimization method of intelligent office automation system | |
| CN108763323A (en) | Meteorological lattice point file application process based on resource set and big data technology | |
| CN102497419B (en) | ESB (Enterprise Service Bus) task scheduling method orienting to large-scale requests and scheduling system | |
| CN104156505B (en) | A kind of Hadoop cluster job scheduling method and devices based on user behavior analysis | |
| CN115794341A (en) | Task scheduling method, device, equipment and storage medium based on artificial intelligence | |
| CN104731606A (en) | System and method based on identification grasping technology to achieve enterprise application integration management | |
| CN109101307A (en) | " a key safety " wechat small routine backstage combination picture processing method based on mobile terminal | |
| CN106802822A (en) | A kind of cloud data center cognitive resources dispatching method based on moth algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |