CN105677763A - Image quality evaluating system based on Hadoop - Google Patents
Image quality evaluating system based on Hadoop Download PDFInfo
- Publication number
- CN105677763A CN105677763A CN201511022591.3A CN201511022591A CN105677763A CN 105677763 A CN105677763 A CN 105677763A CN 201511022591 A CN201511022591 A CN 201511022591A CN 105677763 A CN105677763 A CN 105677763A
- Authority
- CN
- China
- Prior art keywords
- image
- image quality
- quality measure
- mapreduce
- hadoop
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
Abstract
The invention discloses an image quality evaluating system based on Hadoop. The image quality evaluating system comprises a client and a Hadoop cluster; the client comprises a service selection module and a picture transmission module; and the Hadoop cluster comprises a main node and a plurality of sub-nodes. The main node is used for work initialization, work distribution and work coordination and execution, and is used for managing a file system of the cluster; each sub-node is provided with an image quality evaluating module used for executing map tasks and reduce tasks and storing data blocks; and the main node is provided with a communication server which is used for receiving images sent by the client and opens MapReduce work to each user for image quality evaluation. According to the invention, the distributed parallel computing advantage of the Hadoop cluster is employed, time spending on processing quality evaluation of a large number of pictures can be effectively shortened, and the user experience is improved.
Description
Technical field
The present invention relates to image intelligent process field, particularly to a kind of image quality measure system based on Hadoop.
Background technology
In recent years, image quality measure, because of its great demand potential in various application, causes the concern of many scholars. Image quality measure can help people to pick out high artistic image, filters out low artistic image, frees people is worked from irksome image management. Such as, in image retrieval, it is intended that computer also can not only can retrieve image according to picture quality according to picture material.
Nowadays major part scholar is intended to utilize substantial amounts of image study image quality measure model about the work of image quality measure, and image carries out classification and the predicted picture mass fraction of height aesthetic feeling. In order to improve the accuracy of system, people emphasize the importance of image characteristics extraction, including manual feature and local feature. In order to improve accuracy further, setting up huge image data base, the comment of user, geography information etc. are also considered by system. Generally speaking, existing system lacks the attention to image quality measure efficiency and Consumer's Experience.
Along with popularizing of smart mobile phone, people obtain and storage image is increasingly easier, and amount of images is also increasing severely. It is desirable to image quality measure algorithm and can help the image on managing mobile phone. Owing to image quality measure process is complicated, consuming time, on mobile phone, operation image quality evaluation algorithm process speed is slow, inefficiency, particularly when amount of images is huge.
Hadoop is made up of numerous computer nodes, it is achieved that distributed storage and parallel computation, the applicable open source software basic framework processing big data, is widely used in cloud computing. The image procossing that appears as of cloud computing technology provides new thinking. By in conjunction with cloud computing technology, image processing efficiency is greatly improved. University Of Chongqing Zhang little Hong et al. invents the search method (application number: 201310038448.8, publication number: 103207889A) of a kind of magnanimity facial image based on Hadoop. This patent mainly utilizes Hadoop to realize image retrieval function, indexes including to view data, and carries out distributed search and complete the fast search of facial image. Owing to lacking the design of client, the face identification method of this patent of invention may not apply on mobile terminal.
By the calculating resource that high in the clouds is abundant, Hadoop can alleviate the computation burden that terminal is heavy significantly. But, also do not find Hadoop technology to be applied on image quality measure at present both at home and abroad. Additionally, Hadoop is initially designed to the process of big text data, the process for view data needs to do Curve guide impeller again.
Summary of the invention
In order to overcome disadvantages mentioned above and the deficiency of prior art, it is an object of the invention to provide a kind of image quality measure system based on Hadoop, image quality measure algorithm complicated, consuming time in client can be moved to and there is the abundant high in the clouds calculating resource, be exchanged for the high efficiency of image quality measure and good Consumer's Experience by flow.
The purpose of the present invention is achieved through the following technical solutions:
A kind of image quality measure system based on Hadoop, including client and Hadoop cluster;
Described client includes service selecting module and picture transfer module; Described service selecting module is for selecting user to need image and the service of quality evaluation, and shows the result of the image quality measure returned from server end;
Described picture transfer module is for sending user's request, transmission user images by the Internet to server end, and receives the result returned from server end;
Described Hadoop cluster includes host node and multiple child node; Described host node is responsible for the coordination execution of the initialization of operation, the distribution of operation, operation, is responsible for the file system of cluster simultaneously; Described child node is provided with image quality evaluation module, is responsible for execution and the data block storage of map task and reduce task; Described host node is provided with the communication server, and the described communication server is responsible for receiving the image of client transmission and opening a MapReduce operation for each user carrying out image quality measure.
The work process of the described communication server is as follows:
First, the image of each user received is stored under particular category, and this catalogue is uploaded to HDFS; Then, a MapReduce operation is opened for each user; Finally, MapReduce calls the image of the C++ dynamic link library process input of the image quality measure algorithm of image quality evaluation module.
One MapReduce operation of described unlatching carries out image quality measure, particularly as follows:
(1) MapReduce workflow is set up: set up MapReduce and process the workflow of multi-user's request, set up the workflow of the MapReduce many images of process;
(2) defining the input data type of image file: the data type ImgFile class of definition image input, image file pattern of the input ImgFileInputFormat class, image key-value pair reads in form ImgFileRecordReader class;
(3) in map function, image quality measure is realized: realize MapReduce and to the algorithm of image quality measure and realize MapReduce function.
Described set up MapReduce workflow, particularly as follows:
(1-1) workflow of MapReduce parallel processing multi-user request is set up: the data of each user, as the input of a MapReduce operation, realize the parallel processing of many operations by job scheduler;
(1-2) set up MapReduce and process the workflow of many images: whole image is inputted burst as one, and using whole burst as a record; In the map stage, every image of user will be counted as a burst, the map function achieving image quality measure algorithm process; Input picture is represented by the key-value pair form of<image file name, picture material>, and output result is represented by the key-value pair form of<image file name, image quality measure result>; In the reduce stage, the result of map task is collected and exports a text by reduce task.
The input data type of described definition image file, particularly as follows:
(2-1) the data type ImgFile class of image input is defined, image input ImgFile class realizes Writable interface, definition getImage () member function, getHeight () member function, getWidth () member function;
(2-2) image file pattern of the input ImgFileInputFormat class is defined, image file pattern of the input ImgFileInputFormat Similar integral FlieInputFormat class, support ImgFile class, the image file of input is cut into input sliced fashion, increase the definition that image file reads, with a width complete image for a burst, do not carry out file division;
(2-3) define image key-value pair and read in form ImgFileRecordReader class, image key-value pair reads in form ImgFileRecordReader Similar integral RecordReader class, input key-value pair is defined as<image file name, picture material>form, and from InputSplit, read the key-value pair of record for Mapper process; Described image file name is an example of Text type, and described picture material is an example of ImgFile type.
Described in map function, realize image quality measure, particularly as follows:
(3-1) MapReduce calling image quality measure algorithm is realized, namely the MapReduce program write with Java language uses JNI interface interchange by the image quality measure algorithm of C++ code development, realizing classification and the prediction of quality of image, detailed process is:
First, the C++ source file of the core algorithm of image quality measure is generated; Then C++ source file is compiled into so file that can load and call; Finally, the classification trained and assessment models are stored in the data file, calls in execution task for algorithm;
(3-2) realize MapReduce function, namely realize map and reduce function in map class and reduce apoplexy due to endogenous wind, particularly as follows:
In the map stage, first, map function receives an image and reading is ImgFile type, as the picture material in key-value pair<image file name, picture material>; Then, obtain the picture material of key-value pair and convert thereof into int array; Then, the image input being input in JNI function setSourceImageJni () by int array to arrange in C++ code; Finally, then with function Evaluate () or Classify () it is carried out region division, feature extraction and classification assessment process;
In the reduce stage, reduce function collects in classification or Score on Prediction result to the text of every image of map function output.
Described Hadoop cluster uses Hadoop to achieve instrument HadoopArchives and saves cluster host node namenode internal memory, particularly as follows:
First, it was HAR file with HadoopArchives picture archiving before utilizing MapReduce to carry out image quality measure; Then, HAR files passe to HDFS; Finally HAR file is processed as the input of MapReduce.
Adopting the task JVM mode of operation reused when described Hadoop cluster carries out image quality measure, namely each map task can reuse an identical JVM, particularly as follows:
The mapred.job.reuse.jvm.num.tasks parameter in HDOOP_CONF_DIR/mapred-site.xml file be set to-1, JavaVirtualMachine reuse unlimited.
Use FairScheduler job scheduling strategy when described Hadoop cluster carries out image quality measure, calculate resource according to the size of each operation, uplink time schedule job and reasonable distribution, allow All Jobs can obtain the most Fairshare of cluster resource.
Communication between described client and Hadoop cluster adopts socket technology.
Compared with prior art, the present invention has the following advantages and beneficial effect:
(1) the image quality measure system based on Hadoop of the present invention, by complicated image quality measure algorithm is moved to high in the clouds, the application in client becomes simple, becomes light client;
(2) the image quality measure system based on Hadoop of the present invention, user data distributed storage beyond the clouds, efficient parallel process, by utilizing the calculating resource that high in the clouds is enriched and the design optimizing MapReduce, the request of client can quickly be responded, and improves Consumer's Experience.
(3) the image quality measure system based on Hadoop of the present invention, image quality measure algorithm and model can be easily accomplished upgrading at server end, and client need not make any change on hardware and software, applies convenient.
(4) the image quality measure system based on Hadoop of the present invention, picture construction MapReduce framework is processed for Hadoop, achieve the parallel processing of great amount of images, there is good autgmentability, extend to the application such as high-volume image file format conversion, video mode identification.
Accompanying drawing explanation
Fig. 1 is the image quality measure system physical framework based on Hadoop of embodiments of the invention.
Fig. 2 is the image quality measure working-flow figure of embodiments of the invention.
Fig. 3 is the workflow diagram of the image quality evaluation module of embodiments of the invention.
Fig. 4 is the flow chart of the MapReduce operation of embodiments of the invention.
Detailed description of the invention
Below in conjunction with embodiment, the present invention is described in further detail, but embodiments of the present invention are not limited to this.
Embodiment
As it is shown in figure 1, a kind of image quality measure system based on Hadoop, including client and Hadoop cluster; Hadoop cluster includes host node and multiple child node. Communication service runs in client and Hadoop cluster simultaneously, is responsible for transmission data, including from client upload image with return image quality measure result; Described communication service adopts socket technology.
Client include service selecting module and and picture transfer module; Described service selecting module is for selecting user to need image and the service of quality evaluation, and shows the result of the image quality measure returned from server end; Described picture transfer module is for sending user's request, transmission user images by the Internet to server end, and receives the result returned from server end.
As in figure 2 it is shown, host node is responsible for the coordination execution of the initialization of operation, the distribution of operation, operation, it is responsible for the file system of cluster simultaneously; Described child node is provided with image quality evaluation module, is responsible for execution and the data block storage of map task and reduce task; Described host node is provided with the communication server, and the described communication server is responsible for receiving the image sended over from client and opening a MapReduce operation for each user carrying out image quality measure.
As it is shown on figure 3, the workflow of the image quality evaluation module of the present embodiment is as follows:
First, image data base is extracted aesthetic features; Then, trained by machine and Evaluation Model on Quality is set up in study, including image aesthetics grader and image aesthetics regression model; Finally, the image to user's input, utilize the aesthetic-qualitative level grader set up and aesthstic regression model to realize the classification of image aesthetic-qualitative level height and the prediction of aesthetic score.
The work process of the communication server of the present embodiment is as follows:
First, the image of each user received is stored under particular category, and this catalogue is uploaded to HDFS; Then, a MapReduce operation is opened for each user; Finally, MapReduce calls the image of the C++ dynamic link library process input of the image quality measure algorithm of image quality evaluation module.
Described HDFS is Hadoop distributed file system.
In order to utilize Hadoop to realize parallel processing, there is the great amount of images of multi-user, build MapReduce framework and complete image quality measure, including setting up MapReduce workflow, defining the input data type of image file and realize image quality measure algorithm in map function. Particularly as follows:
(1) set up MapReduce workflow, namely set up MapReduce and process the workflow of multi-user's request, set up the workflow of the MapReduce many images of process;
(1-1) setting up the workflow of MapReduce parallel processing multi-user request, namely the data of each user are as the input of a MapReduce operation, realized the parallel processing of many operations by job scheduler;
(1-2) set up MapReduce and process the workflow of many images, as shown in Figure 4. Whole image is inputted burst as one, and using whole burst as a record. In the map stage, every image of user will be counted as a burst, the map function achieving image quality measure algorithm process. Input picture is represented by the key-value pair form of<image file name, picture material>, and output result is represented by the key-value pair form of<image file name, image quality measure result>. In the reduce stage, the result of map task is collected and exports a text by reduce task.
(2) defining the input data type of image file, i.e. the data type ImgFile class of definition image input, image file pattern of the input ImgFileInputFormat class, image key-value pair reads in form ImgFileRecordReader class. Particularly as follows:
(2-1) the data type ImgFile class of image input is defined, image input ImgFile class realizes Writable interface, define a series of member functions such as getImage (), getHeight (), andgetWidth ();
(2-2) image file pattern of the input ImgFileInputFormat class is defined, image file pattern of the input ImgFileInputFormat Similar integral FlieInputFormat class, support ImgFile class, the image file of input is cut into input sliced fashion, add the definition that image file reads, with a width complete image for a burst, do not carry out file division;
(2-3) define image key-value pair and read in form ImgFileRecordReader class, image key-value pair reads in form ImgFileRecordReader Similar integral RecordReader class, input key-value pair is defined as<image file name, picture material>form, and from InputSplit, read the key-value pair of record for Mapper process. Described image file name is an example of Text type, and described picture material is an example of ImgFile type.
(3) in map function, realize image quality measure algorithm, namely realize MapReduce calling and realizing MapReduce function image quality measure algorithm. Particularly as follows:
(3-1) realizing MapReduce calling image quality evaluation module, the MapReduce program namely write with Java language uses JNI interface interchange by the image quality measure algorithm of C++ code development, it is achieved the classification of image and Score on Prediction. Particularly as follows:
First, the C++ source file of the core algorithm of image quality measure is generated; Then C++ source file is compiled into so file that can load and call; Finally, the classification trained and assessment models are stored in the data file, calls in execution task for algorithm.
The described image quality measure system based on Hadoop uses Hadoop file cache instrument HadoopDistributedCache so file and classification and the assessment models data file trained to be deployed in each node in cluster.
(3-2) realize MapReduce function, namely realize map and reduce function in map class and reduce apoplexy due to endogenous wind, particularly as follows:
In the map stage, first, map function receives an image and reading is ImgFile type, as the picture material in key-value pair<image file name, picture material>; Then, obtain the picture material of key-value pair and convert thereof into int array; Then, the image input being input in JNI function setSourceImageJni () by int array to arrange in C++ code; Finally, then with function Evaluate () or Classify (), it is carried out region division, feature extraction and classification assessment etc. and process.
In the reduce stage, reduce function collects in classification or Score on Prediction result to the text of every image of map function output.
Hadoop performance in processing large amount of small documents is not good enough, processes the application of a large amount of little image files to adapt to Hadoop cluster, improves systematic function, Hadoop cluster is optimized, and reuses including saving internal memory and task JVM.
Described saving internal memory, namely uses Hadoop to achieve instrument HadoopArchives and saves cluster host node namenode internal memory, particularly as follows:
First, it was HAR file with HadoopArchives picture archiving before utilizing MapReduce to carry out image quality measure; Then, HAR files passe to HDFS; Finally HAR file is processed as the input of MapReduce.
Described task JVM reuses, and namely each map task can reuse an identical JVM. Particularly as follows:
The mapred.job.reuse.jvm.num.tasks parameter in HDOOP_CONF_DIR/mapred-site.xml file be set to-1, JavaVirtualMachine can reuse unlimited.
Hadoop acquiescence uses FIFO scheduler, asks for concurrent multi-user's different images number, and systematic function is not high. In order to make the operation of each user can obtain the most Fairshare of cluster resource, improve Consumer's Experience, the present embodiment uses FairScheduler job scheduling strategy, calculates resource according to the schedule jobs such as the size of each operation, uplink time and reasonable distribution, particularly as follows:
Mapred.jobtracker.taskScheduler parameter in HADOOP_CONF_DIR/mapred-site.xml file is arranged to org.apache.hadoop.mapred.FairScheduler. It addition, mapred.fairscheduler.sizebasedweight parameter is arranged to true, when opening after this option, system can using job size one of determiner calculating resource as distribution.
In the present embodiment, the present embodiment builds Hadoop cluster on the Dell precisionT5610 work station virtual machine VMwareWorkstation11 have 64G internal memory, cluster is made up of 1 host node and 5 child nodes, and the basic software constituting each node includes: Ubuntu10.04LTS operating system, Hadoop0.21.0, JRE1.6.0_33, OpenCV-2.2.0. The configuration of unit is identical with each node of cluster. The present embodiment is using 10 groups of quantity respectively image of 100 to 1000 as input, and the process time is that operation starts to operation to have processed. Unit and Hadoop cluster different images number input processing time to such as table 1:
Table 1Hadoop cluster and unit process time contrast
| 100 | 200 | 300 | 400 | 500 | 600 | 700 | 800 | 900 | 1000 | |
| Unit (second) | 10.01 | 23.30 | 34.69 | 43.16 | 52.76 | 61.39 | 70.23 | 79.77 | 90.70 | 100.70 |
| Cluster (second) | 1.83 | 3.62 | 5.15 | 6.30 | 7.77 | 9.10 | 10.43 | 11.93 | 13.30 | 15.03 |
Pass through above table, it can be seen that the process time of unit is longer than the time of Hadoop cluster a lot. When amount of images is more big, the time of unit process operation is greatly increased, and the advantage of cluster becomes apparent from. Visible, the application of Hadoop cluster makes the efficiency of image quality measure system significantly improve.
The present embodiment, by simulating the request of concurrent user's different images number, verifies the scheduling strategy of FairScheduler. Concurrent user number is 3, respectively user 1, user 2 and user 3, the picture number inputted respectively 5,40 and 500. The scheduling strategy of system default FIFO, the operation deadline that therefore the different requests order of concurrent user is corresponding by affecting each user. The order of can request that of 3 operations is 1-2-3,1-3-2,2-1-3,2-3-1,3-1-2 and 3-2-1. The present embodiment calculates average handling time and the standard deviation of the time of process of each user 6 request orders under FIFO and FairScheduler scheduling strategy. Contrast under two kinds of scheduling strategies of FIFO and FairScheduler is as shown in table 3:
The contrast of table 2FairScheduler and FIFO scheduling strategy
By upper table it follows that first, when system process images quantity is less, the average used time using the cost of FairScheduler scheduling strategy is fewer than FIFO. FairScheduler scheduling strategy is effectively shortened the waiting time of operation so that it is the time spent is greatly reduced than FIFO. One short operation will complete within the rational time, even if the long working of another user is also in running. Second, use user's used time standard deviation of FairScheduler scheduling strategy to be respectively less than the standard deviation using FIFO scheduling strategy. Obviously, when the job sequence that use FIFO scheduling strategy is different, the waiting time of user is quite different, and FairScheduler is relatively stable. Visible, cluster resource is shared in the use of FairScheduler scheduling strategy with allowing each user fairness, improves the experience of user.
Above-described embodiment is the present invention preferably embodiment; but embodiments of the present invention are also not restricted by the embodiments; the change made under other any spirit without departing from the present invention and principle, modification, replacement, combination, simplification; all should be the substitute mode of equivalence, be included within protection scope of the present invention.
Claims (10)
1. the image quality measure system based on Hadoop, it is characterised in that include client and Hadoop cluster;
Described client includes service selecting module and picture transfer module; Described service selecting module is for selecting user to need image and the service of quality evaluation, and shows the result of the image quality measure returned from server end; Described picture transfer module is for sending user's request, transmission user images by the Internet to server end, and receives the result returned from server end;
Described Hadoop cluster includes host node and multiple child node; Described host node is responsible for the coordination execution of the initialization of operation, the distribution of operation, operation, is responsible for the file system of cluster simultaneously; Described child node is provided with image quality evaluation module, is responsible for execution and the data block storage of map task and reduce task; Described host node is provided with the communication server, and the described communication server is responsible for receiving the image of client transmission and opening a MapReduce operation for each user carrying out image quality measure.
2. the image quality measure system based on Hadoop according to claim 1, it is characterised in that the work process of the described communication server is as follows:
First, the image of each user received is stored under particular category, and this catalogue is uploaded to HDFS; Then, a MapReduce operation is opened for each user; Finally, MapReduce calls the image of the C++ dynamic link library process input of the image quality measure algorithm of image quality evaluation module.
3. the image quality measure system based on Hadoop according to claim 2, it is characterised in that described one MapReduce operation of unlatching carries out image quality measure, particularly as follows:
(1) MapReduce workflow is set up: set up MapReduce and process the workflow of multi-user's request, set up the workflow of the MapReduce many images of process;
(2) defining the input data type of image file: the data type ImgFile class of definition image input, image file pattern of the input ImgFileInputFormat class, image key-value pair reads in form ImgFileRecordReader class;
(3) in map function, image quality measure is realized: realize MapReduce and to the algorithm of image quality evaluation module and realize MapReduce function.
4. the image quality measure system based on Hadoop according to claim 3, it is characterised in that described set up MapReduce workflow, particularly as follows:
(1-1) workflow of MapReduce parallel processing multi-user request is set up: the data of each user, as the input of a MapReduce operation, realize the parallel processing of many operations by job scheduler;
(1-2) set up MapReduce and process the workflow of many images: whole image is inputted burst as one, and using whole burst as a record; In the map stage, every image of user will be counted as a burst, the map function achieving image quality measure algorithm process; Input picture is represented by the key-value pair form of<image file name, picture material>, and output result is represented by the key-value pair form of<image file name, image quality measure result>; In the reduce stage, the result of map task is collected and exports a text by reduce task.
5. the image quality measure system based on Hadoop according to claim 3, it is characterised in that the input data type of described definition image file, particularly as follows:
(2-1) the data type ImgFile class of image input is defined, image input ImgFile class realizes Writable interface, definition getImage () member function, getHeight () member function, getWidth () member function;
(2-2) image file pattern of the input ImgFileInputFormat class is defined, image file pattern of the input ImgFileInputFormat Similar integral FlieInputFormat class, support ImgFile class, the image file of input is cut into input sliced fashion, increase the definition that image file reads, with a width complete image for a burst, do not carry out file division;
(2-3) define image key-value pair and read in form ImgFileRecordReader class, image key-value pair reads in form ImgFileRecordReader Similar integral RecordReader class, input key-value pair is defined as<image file name, picture material>form, and from InputSplit, read the key-value pair of record for Mapper process; Described image file name is an example of Text type, and described picture material is an example of ImgFile type.
6. the image quality measure system based on Hadoop according to claim 3, it is characterised in that described realize image quality measure in map function, particularly as follows:
(3-1) MapReduce calling image quality measure algorithm is realized, namely the MapReduce program write with Java language uses JNI interface interchange by the image quality measure algorithm of C++ code development, realizing classification and the prediction of quality of image, detailed process is:
First, the C++ source file of the core algorithm of image quality measure is generated; Then C++ source file is compiled into so file that can load and call; Finally, the classification trained and assessment models are stored in the data file, calls in execution task for algorithm;
(3-2) realize MapReduce function, namely realize map and reduce function in map class and reduce apoplexy due to endogenous wind, particularly as follows:
In the map stage, first, map function receives an image and reading is ImgFile type, as the picture material in key-value pair<image file name, picture material>; Then, obtain the picture material of key-value pair and convert thereof into int array; Then, the image input being input in JNI function setSourceImageJni () by int array to arrange in C++ code; Finally, then with function Evaluate () or Classify () it is carried out region division, feature extraction and classification assessment process;
In the reduce stage, reduce function collects in classification or Score on Prediction result to the text of every image of map function output.
7. the image quality measure system based on Hadoop according to claim 1, it is characterised in that described Hadoop cluster uses Hadoop to achieve instrument HadoopArchives and saves cluster host node namenode internal memory, particularly as follows:
First, it was HAR file with HadoopArchives picture archiving before utilizing MapReduce to carry out image quality measure; Then, HAR files passe to HDFS; Finally HAR file is processed as the input of MapReduce.
8. the image quality measure system based on Hadoop according to claim 1, it is characterized in that, adopting the task JVM mode of operation reused when described Hadoop cluster carries out image quality measure, namely each map task can reuse an identical JVM, particularly as follows:
The mapred.job.reuse.jvm.num.tasks parameter in HDOOP_CONF_DIR/mapred-site.xml file be set to-1, JavaVirtualMachine reuse unlimited.
9. the image quality measure system based on Hadoop according to claim 1, it is characterized in that, described Hadoop cluster uses FairScheduler job scheduling strategy when carrying out image quality measure, calculate resource according to the size of each operation, uplink time schedule job and reasonable distribution, allow All Jobs can obtain the most Fairshare of cluster resource.
10. the image quality measure system based on Hadoop according to claim 1, it is characterised in that the communication between described client and Hadoop cluster adopts socket technology.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511022591.3A CN105677763B (en) | 2015-12-29 | 2015-12-29 | A kind of image quality measure system based on Hadoop |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201511022591.3A CN105677763B (en) | 2015-12-29 | 2015-12-29 | A kind of image quality measure system based on Hadoop |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105677763A true CN105677763A (en) | 2016-06-15 |
| CN105677763B CN105677763B (en) | 2019-08-20 |
Family
ID=56298092
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201511022591.3A Active CN105677763B (en) | 2015-12-29 | 2015-12-29 | A kind of image quality measure system based on Hadoop |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105677763B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126601A (en) * | 2016-06-20 | 2016-11-16 | 华南理工大学 | A kind of social security distributed preprocess method of big data and system |
| CN106371908A (en) * | 2016-08-31 | 2017-02-01 | 武汉鸿瑞达信息技术有限公司 | Optimization method for image/video filter task distribution based on PSO (Particle Swarm Optimization) |
| CN106549949A (en) * | 2016-10-31 | 2017-03-29 | 广西东方道迩科技有限公司 | A kind of image data processing system and its image processing method |
| CN108256118A (en) * | 2018-02-13 | 2018-07-06 | 腾讯科技（深圳）有限公司 | Data processing method, device, system, computing device and storage medium |
| CN108900335A (en) * | 2018-06-28 | 2018-11-27 | 泰康保险集团股份有限公司 | Workflow management method and device based on Hadoop |
| CN110378332A (en) * | 2019-06-14 | 2019-10-25 | 上海咪啰信息科技有限公司 | A kind of container terminal case number (CN) and Train number recognition method and system |
| CN113836130A (en) * | 2021-09-28 | 2021-12-24 | 深圳创维智慧科技有限公司 | Data quality evaluation method, device, equipment and storage medium |
| CN115422126A (en) * | 2022-11-04 | 2022-12-02 | 浪潮软件股份有限公司 | Method, system and device for rapidly transferring certificate OFD format file to picture |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI779808B (en) * | 2021-08-30 | 2022-10-01 | 宏碁股份有限公司 | Image processing method |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103207889A (en) * | 2013-01-31 | 2013-07-17 | 重庆大学 | Method for retrieving massive face images based on Hadoop |
| US20140270494A1 (en) * | 2013-03-15 | 2014-09-18 | Sri International | Computer vision as a service |
-
2015
- 2015-12-29 CN CN201511022591.3A patent/CN105677763B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103207889A (en) * | 2013-01-31 | 2013-07-17 | 重庆大学 | Method for retrieving massive face images based on Hadoop |
| US20140270494A1 (en) * | 2013-03-15 | 2014-09-18 | Sri International | Computer vision as a service |
Non-Patent Citations (2)
| Title |
|---|
| WEINING WANG 等: "A System of Image Aesthetic Classification and Evaluation Using Cloud Computing", 《CCPR 2014》 * |
| 王贤伟: "基于Hadoop的外观专利图像检索系统的研究与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106126601A (en) * | 2016-06-20 | 2016-11-16 | 华南理工大学 | A kind of social security distributed preprocess method of big data and system |
| CN106371908A (en) * | 2016-08-31 | 2017-02-01 | 武汉鸿瑞达信息技术有限公司 | Optimization method for image/video filter task distribution based on PSO (Particle Swarm Optimization) |
| CN106549949A (en) * | 2016-10-31 | 2017-03-29 | 广西东方道迩科技有限公司 | A kind of image data processing system and its image processing method |
| CN108256118A (en) * | 2018-02-13 | 2018-07-06 | 腾讯科技（深圳）有限公司 | Data processing method, device, system, computing device and storage medium |
| CN108256118B (en) * | 2018-02-13 | 2023-09-22 | 腾讯科技（深圳）有限公司 | Data processing method, device, system, computing equipment and storage medium |
| CN108900335A (en) * | 2018-06-28 | 2018-11-27 | 泰康保险集团股份有限公司 | Workflow management method and device based on Hadoop |
| CN110378332A (en) * | 2019-06-14 | 2019-10-25 | 上海咪啰信息科技有限公司 | A kind of container terminal case number (CN) and Train number recognition method and system |
| CN113836130A (en) * | 2021-09-28 | 2021-12-24 | 深圳创维智慧科技有限公司 | Data quality evaluation method, device, equipment and storage medium |
| CN115422126A (en) * | 2022-11-04 | 2022-12-02 | 浪潮软件股份有限公司 | Method, system and device for rapidly transferring certificate OFD format file to picture |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105677763B (en) | 2019-08-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105677763A (en) | Image quality evaluating system based on Hadoop | |
| US11455189B2 (en) | Task scheduling simulation system | |
| CN107515878B (en) | Data index management method and device | |
| CN105045607A (en) | Method for achieving uniform interface of multiple big data calculation frames | |
| CN105550268A (en) | Big data process modeling analysis engine | |
| CN102750406A (en) | Multi-version management method for model set and difference model-based power grid model | |
| CN106126601A (en) | A kind of social security distributed preprocess method of big data and system | |
| CN110968579B (en) | Execution plan generation and execution method, database engine and storage medium | |
| CN102929646B (en) | Application program generation method and device | |
| CN107291539B (en) | Cluster program scheduler method based on resource significance level | |
| CN103699656A (en) | GPU-based mass-multimedia-data-oriented MapReduce platform | |
| CN113407649A (en) | Data warehouse modeling method and device, electronic equipment and storage medium | |
| CN102508919A (en) | Data processing method and system | |
| CN110471754A (en) | Method for exhibiting data, device, equipment and storage medium in job scheduling | |
| Wang et al. | An efficient image aesthetic analysis system using Hadoop | |
| CN108763323A (en) | Meteorological lattice point file application process based on resource set and big data technology | |
| CN108287889B (en) | A kind of multi-source heterogeneous date storage method and system based on elastic table model | |
| CN112182031B (en) | Data query method and device, storage medium and electronic device | |
| CN108319604B (en) | Optimization method for association of large and small tables in hive | |
| CN113010542A (en) | Service data processing method and device, computer equipment and storage medium | |
| CN116089414B (en) | Time sequence database writing performance optimization method and device based on mass data scene | |
| CN111190704A (en) | Task classification processing method based on big data processing framework | |
| CN105912621A (en) | Area building energy consumption platform data storing and query method | |
| CN104658020A (en) | Novel media production and management system | |
| CN104077398B (en) | Job assignment system and method based on environment of gathering together Hadoop more |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |