CN103294550A - Heterogeneous multi-core thread scheduling method, heterogeneous multi-core thread scheduling system and heterogeneous multi-core processor - Google Patents

Abstract

The invention relates to a heterogeneous multi-core thread scheduling method, which includes generating sorting lists for threads and cores respectively according to the dynamic characteristics of programs, and finding out the optimal stable matching between threads and cores according to the sorting lists, and performing thread scheduling according to the stable matching . Including receiving the eigenvector of the thread running on the core, and selecting a priority for each core for the thread; sorting each thread for each core; receiving the sorted list of each thread and core, and finding out the thread and The stable matching result of the core; receive the matching result, schedule through the operating system, and assign each thread to the corresponding core to run. Avoiding the huge overhead caused by sampling scheduling; taking into account more complex factors that affect performance and power consumption, only the relative relationship of prediction is needed instead of specific values, which reduces the complexity of the model and improves the accuracy of scheduling .

Description

A heterogeneous multi-core thread scheduling method, system and heterogeneous multi-core processor

technical field

The present invention relates to the field of a thread scheduling method (threads scheduling policy) under the condition that the number of threads and cores in Single-ISA heterogeneous multi-core processors (Single-ISA heterogeneous multi-core processors) is equal, and in particular relates to a method based on threads and checking After selecting and prioritizing each other, the implementation of the thread scheduling method is completed with the Gale-Shapley algorithm.

Background technique

With the development of integrated circuit technology, more and more cores are integrated into the same system-on-chip, and chip multi-processors (CMP) have gradually become a mainstream processor architecture. On-chip multi-core processors provide better performance for programs running in parallel in the system by integrating multiple identical general-purpose cores on the chip, but they are also limited by power consumption, heat dissipation, and chip area. In order to make full use of the limited power consumption and area on the chip more effectively, the industry and academia have proposed a heterogeneous multi-core processor structure.

Heterogeneous multi-core processors have various forms, and the present invention mainly relates to single-instruction-set heterogeneous multi-core processors (Single-ISA heterogeneous multi-core processors). In a single instruction set heterogeneous multi-core processor, different types of cores share the same instruction set. Differences between cores can be caused by parameters such as frequency, cache size, power budget, etc., or by basic structural design (eg: out-of-order or in-order, instruction issue width, etc. ) caused by the difference. In addition, the present invention is mainly aimed at the situation that each core in the heterogeneous multi-core processor runs a single-threaded program, so the number of threads is always equal to the number of cores in the system, and the threads can be regarded as equivalent to the program.

Different programs often have different program characteristics. Furthermore, even for the same program, its program characteristics will change significantly according to the change of input set and execution stage.

In a heterogeneous multi-core processor, each thread is scheduled to run on the most suitable core according to the program characteristics, which is called thread scheduling. The purpose of thread scheduling is to use appropriate cores to provide better performance for threads, and at the same time avoid waste of power consumption as much as possible, so that the limited power consumption and area resources on the chip can be used more effectively.

Scheduling methods can be divided into static and dynamic. Among them, the static scheduling method guesses the performance of each thread running on different types of cores by extracting the characteristics of the program offline and has nothing to do with the specific execution environment. According to the prediction results, each thread is scheduled to the run on the corresponding core. The static scheduling method only uses the differences between programs, ignoring the different program characteristics of the program itself in different execution stages, so the static scheduling method has natural defects.

The sampling-based dynamic scheduling method divides the scheduling into two phases: the sampling phase and the stable execution phase. When a trigger event that marks a significant change in the dynamic characteristics of the program occurs, it enters the sampling phase; in the sampling phase, each thread is scheduled to each type of core for trial operation, so it is necessary to traverse all scheduling schemes and record Next, the corresponding performance of each scheduling scheme is selected; then the scheduling scheme with the best performance is selected to enter the stable execution stage for a long period of time until the next trigger event occurs. The sampling-based dynamic scheduling method can make full use of the dynamic characteristics of the program for scheduling. However, the sampling phase will bring a lot of thread migration costs, and when traversing different scheduling schemes, the program needs to be tested under various non-ideal scheduling schemes, resulting in a very large performance overhead; in addition, the sampling overhead will be As the types of cores in the system increase rapidly, this kind of scheduling method has poor scalability and cannot be applied in practice.

In order to avoid the overhead caused by sampling, a class of scheduling methods based on heuristics is proposed. This type of scheduling method uses some hardware monitoring components (monitors) to collect some key information in the running of the program, such as IPC, cache failure rate, blocking time, etc., and uses these dynamic information to estimate the different types of threads according to empirical rules. The performance of running on the core, and then use the greedy algorithm to select the appropriate core for the thread according to the size of the income.

The following is a brief introduction to representative technical solutions in this type of scheduling method:

In a heterogeneous multi-core processor composed of cores with different frequencies, the threads are sorted according to the IPC of the previous execution stage from high to low, and the cores are sorted by frequency, and then the threads and cores are sorted according to their relative positions. match. In a similar way, threads can be divided into two types: compute-intensive and memory-intensive by collecting information such as the cache miss rate of the thread, and then the compute-intensive Threads are scheduled to run on large cores (for example: high frequency, large cache area, out-of-order execution, etc.), and memory-intensive threads are scheduled to run on small cores (for example: low frequency, small cache area, sequential execution, etc. )run. The starting point of this scheduling method is to allocate computationally intensive threads with high instruction-level parallelism (ILP) to large cores to achieve better performance, and allocate memory-intensive threads to small cores to save power consumption. . A further improvement of this approach is to combine the collected cache failure rate, stall time and other information with the structural parameters of each core to estimate the performance of each thread running on different cores, and then use the greedy algorithm The benefit size schedules threads to run on each core.

This type of scheduling method usually only uses a few important program characteristics (such as cache failure rate, IPC, etc.) and core structural parameters (such as frequency, cache size, etc.) combined with certain domain knowledge or empirical rules to estimate the performance of the program , but in fact, the performance of the program is related to a large number of complex factors, which makes the predictions often not accurate enough, which makes the effect of this kind of scheduling method unsatisfactory.

In addition, most of the existing scheduling methods rely on considering a limited number of factors through a formula model to predict the performance of each thread running on different types of cores. But the actual performance of the program is related to a variety of complex factors, resulting in limited accuracy of such predictions. On the other hand, even if an accurate predictive model exists, its implementation complexity is usually high and does not necessarily lead to better scheduling. For example, suppose the actual performance of a thread running on two different types of cores is (5, 4.8), model A predicts (4.9, 5.1), and model B predicts (10, 1). Obviously, the prediction of model A is more accurate, but the scheduling scheme based on the prediction of model B is more reliable. It can be seen from this example that in fact, what is needed is not the exact value of the performance of threads running on different cores, but a relative relationship, that is, a performance ranking of predicted threads running on each core.

On the other hand, most of the existing scheduling methods start from the thread point of view, take the thread as the decision-making subject, and perform greedy scheduling according to a single optimization goal.

In general, the previously proposed scheduling methods set an optimization goal from the perspective of the program, and use the program as the decision-making body to select the appropriate core. The problem with this one-way selection scheduling method is that during the scheduling process, the core does not have the right to actively decide whether to receive a thread according to its own structural characteristics and power consumption constraints. For example, when a core is selected by a computationally intensive thread, it means that the core can provide the best performance for the thread; but from the perspective of the core, if the core receives the thread, it may cause other If the power consumption exceeds the power budget, this scheduling scheme is obviously not ideal.

Contents of the invention

In order to solve the above-mentioned technical problems, the object of the present invention is to propose a thread scheduling method and a scheduling system for heterogeneous multi-core processors based on the Gale-Shapley algorithm. For the thread scheduling problem in heterogeneous multi-core processors, the present invention can Dynamic scheduling of changes in characteristics effectively avoids the huge overhead brought by the sampling-based scheduling method, and the heuristic scheduling method is difficult to accurately predict the performance of the flaws that lead to unsatisfactory scheduling, and both threads and cores are used as decision-making participants. During the scheduling process, the needs of threads and cores can be taken into account at the same time.

Specifically, the present invention discloses a heterogeneous multi-core thread scheduling method, including generating sorting lists for threads and cores according to the dynamic characteristics of the program, and finding the optimal stable matching of threads and cores according to the sorting lists, according to the Stable matching for thread scheduling.

The thread and core generating sorting lists include generating a sorting model, which specifically includes the following steps:

(1) Choose an ideal database;

(2) extract program sampling fragments from the database;

(3) Run the program sampling fragments on the simulators of each core respectively, and get corresponding responses, and divide the program sampling fragments and their responses into two parts: training set and test set;

(4) Select an appropriate learning algorithm to train the ranking model;

(5) When the test error of the ranking model meets the requirements, the training phase ends.

The said program sampling segment includes feature vectors, for thread, input the feature vector of a program sampling segment, and output a sorted list to each core; for core, input the feature vector of each thread program sampling segment, output as each A sorted list of checkpoints for each thread.

The heterogeneous multi-core thread scheduling method specifically includes the following steps:

Collect all kinds of dynamic information in the running of the thread, and output the feature vector of a program sampling segment of the thread;

Receive the eigenvector of the thread running on the core, and select a priority order for each core for the thread according to it;

Sort each thread for each check;

Receive the sorted list of each thread and core, and find out the stable matching result of thread and core;

After receiving the matching result, scheduling is performed through the operating system, and each thread is assigned to a corresponding core to run.

In the heterogeneous multi-core thread scheduling method, finding out the stable matching of threads and cores includes the following steps:

(1) The thread makes a matching request to the core from high to low according to its priority order. If the core has no matching object, it chooses to accept the request and form a matching pair with it;

(2) If the core already has a matching object, compare the priority of the new thread with the matching object. If the priority of the new thread is higher than the previously accepted thread, choose to accept the new thread as the matching object. If the new thread's If the priority is lower than the previously accepted thread, new requests are rejected;

(3) The rejected thread re-selects the next core in the sorted list to make a matching request until all threads and cores have found matching objects.

Said finding out the stable matching between thread and core includes adopting Gale-Shapley algorithm.

The invention also discloses a heterogeneous multi-core thread scheduling system, which is characterized in that it includes an information collection module, a T sorter, a C sorter, a matcher, and a thread scheduler, wherein:

The information collection module is used to collect various dynamic information in the operation of each thread, and the output is a feature vector of a certain program sampling segment of each thread;

The T sorter is used to receive the feature vectors of the threads running on the core, and to select and prioritize each core for the thread according to it;

C sorter for sorting each thread for each core;

A matcher is used to receive a sorted list of each thread and each core, and obtain a stable matching result of the thread and the core;

The thread scheduler receives the matching result, performs scheduling through the operating system, and assigns each thread to a corresponding core to run.

The invention also discloses a heterogeneous multi-core processor adopting any one of the above heterogeneous multi-core thread scheduling methods.

The invention also discloses a heterogeneous multi-core processor comprising the above-mentioned heterogeneous multi-core thread scheduling system.

The beneficial effects of the present invention are: on the basis of being able to utilize the dynamic characteristics of the program, the huge overhead caused by sampling scheduling is avoided; a nonlinear learning sorting model is used to replace the practice of predicting performance and power consumption with empirical formulas, and more Multiple complex factors that affect performance and power consumption are taken into account, and only the relative relationship of prediction is required rather than specific values, which reduces the complexity of the model and improves the accuracy of scheduling; in the thread scheduling process, by Both the core and the core are regarded as independent decision-making subjects in the game process, so as to take into account the performance requirements of the program and the power consumption limit of the core; use the Gale-Shapley algorithm to find a stable match in Pareto optimality and perform thread scheduling according to it .

Description of drawings

Figure 1 The off-line training framework of the sorting model of the present invention

Fig. 2 structure embodiment of quad-core heterogeneous multi-core processor of the present invention

Detailed ways

The present invention draws lessons from game theory and regards threads and cores as selfish decision-making participants. They will try to maximize their performance or power consumption benefits from their own perspectives, objectively enabling the scheduling method to take into account both threads and cores. Optimizing the objective to get a better global scheduling decision.

In the present invention, it is necessary to obtain the selection priority ranking of threads for each core. And from the perspective of the core, each thread is prioritized for receiving.

In order to obtain the above priority rankings, a ranking model (ranker) needs to be trained using the learn-to-rank technique. As shown in Fig. 1, a specific way that the present invention obtains the ranking model is shown:

Application database Application database: an infinite ideal database containing all programs;

Program sampling segment Sample application phase: The program sampling segment extracted from some sample programs, its program features should be able to represent most of the commonly used programs, and some commonly used program analysis tools such as mika can extract the feature vector of the program;

Simulator Simulator: For a heterogeneous multi-core processor, the number of cores and the type of each core have been determined in advance, and the program sampling fragments are run on the simulators of each core respectively, and corresponding responses (each core response) are obtained, and the program The sampled fragments and their responses are divided into two parts: training set and test set;

Learning algorithm Learning algorithm: The training of the ranking model ranker is a supervised learning process, and an appropriate learning algorithm such as RankBoost is selected according to the situation to train the ranking model;

Ranking model Ranker model: When the test error of the ranking model can meet the requirements, the training phase ends.

For threads, the input of T-ranker is a feature vector of a program fragment, and the output is a sorted list of each core. For T-ranker, the cores to be sorted are fixed, and the input variable is just each The eigenvectors of program fragments, so only one T-ranker needs to be trained to be common on all cores; for a certain core, the input of C-ranker is the eigenvector of each thread fragment, and the output is a For the sorting list, for C-ranker, the threads that need to be sorted are changing, and each core has different structural configuration characteristics, even for the same group of threads, the sorting result pages are different, so it needs to be trained separately for each core A C-ranker.

After the training is completed, the sorting model can be implemented by hardware and integrated on each core. Or as part of the scheduling system of the present invention.

After using the sorting model to obtain their respective sorting lists for threads and cores, a stable match is found according to the Gale-Shapley algorithm, and then thread scheduling is performed according to the matching results. So as to achieve a Pareto optimal state, so that all threads and cores are in a relatively satisfactory state, so as to objectively achieve an approximate global optimal thread scheduling.

Assuming that there are N elements in each set A and set B, and each element has its own prioritized list containing all the elements of the other set, then according to the Gale-Shapley algorithm, a stable The matching state of each element can find the best matching object it can find. An unstable match means that there is a state in which element a in set A and element b in set B each have priority in each other's sorted list than their current matching objects, so a and b are more inclined to Reject their current match and match against each other. A match without unstable factors is a stable match. For sets A and B, there may be multiple stable matches. Theory proves that the matching found according to the Gale-Shapley algorithm is always in the Pareto optimal state and is the best of all stable matchings.

The Gale-Shapley algorithm-based thread scheduling method for heterogeneous multi-core processors provided by the present invention is described by taking a 4-core heterogeneous multi-core processor as an example. Obviously, the present invention can also be extended to a heterogeneous multi-core processor integrated with more cores, and there is no limitation on the type of cores.

As shown in Figure 2, in a 4-core heterogeneous multi-core processor, in addition to 4 cores, it includes the following components: information collection module Monitor on each core, T-ranker T-ranker on each core , a C sorter C-ranker, a matcher Matchmaker, a thread scheduler Scheduler.

Monitor: Used to collect various dynamic information during thread running, including but not limited to cache failure rate, blocking time, number of integer instructions, number of floating point instructions, etc., and its output is the feature vector of a certain program segment of the thread;

T-ranker: Receives the eigenvector of the thread running on the core, and selects and prioritizes each core for the thread according to it, usually based on performance as the sorting standard;

C-ranker: In fact, four sorting models are integrated inside C-ranker, which are used to sort the four threads of each core respectively. The sorting standard can be set according to the performance function under the premise of meeting the power budget. The consumption ratio is sorted from high to low; since each individual ranker needs to receive eigenvectors from four threads, concentrating them together can reduce communication overhead, and only need to receive information once from the four-core Monitor;

Matchmaker: Receives the sorted list of each thread and core, and finds a stable matching result according to the Gale-Shapley algorithm;

Scheduler: Receive the matching result of Matchmaker, schedule through the operating system, and assign each thread to the corresponding core to run.

In order to make the purpose, technical solution and advantages of the present invention clearer, the thread scheduling method for heterogeneous multi-core processors based on the Gale-Shapley algorithm of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The thread scheduling method of the heterogeneous multi-core processor based on the Gale-Shapley algorithm in the embodiment of the present invention includes generating sorting lists for the threads and cores according to the dynamic characteristics of the program, and finding an optimal one using the Gale-Shapley algorithm according to the sorting results. Stable matching for thread scheduling. In this embodiment, it is assumed that there are only four heterogeneous cores core0, core1, core2 and core3 in the system, each of which has a different structural configuration. Obviously, the present invention can also be extended to heterogeneous processors containing more cores, and its implementation is not much different from the quad-core heterogeneous multi-core processor in this example, so it will not be described in detail here, but all should be regarded as to be included within the scope of the present invention.

The following is an example of the ranking model training architecture shown in Figure 1 to introduce the implementation process of the offline training of the ranker model.

First, use a representative sample program such as SPEC2006 as a program library, divide it into a series of program segments according to certain rules, for example, treat every ten million instructions as a program segment; use program analysis tools such as mika, etc. Extract the feature vector of the program segment, which can contain various information such as ILP, number of integer instructions, number of floating-point instructions, cache failure rate, etc.; randomly select some program segments from the library, respectively in the four cores of core0, core1, core2, and core3 Simulate on the simulator, and obtain the corresponding performance information such as IPC, and power consumption information such as performance and power consumption ratio; the randomly selected program segments and their simulation results are randomly divided into two parts: training set and test set. A fixed learning ranking algorithm such as RankBoost is used to train the ranking model, and the training process of the ranking model is a supervised learning process.

For T-ranker, its input is the eigenvector of a program segment, and the output is the performance ranking of the same program segment running on four cores, that is, the priority ordering of threads for core selection. As mentioned above, T-ranker only One model needs to be trained and can be used for four cores respectively; for a C-ranker of a certain core, the input is the feature vector of four different program segments running on the core, and the output is the ranking of the four performance-to-power ratios, That is, to check the acceptance priority sorting of threads, C-ranker needs to train an independent sorting model for each core separately. The training phase of the model ends when the test error of the model on the test set is acceptably low.

After the training of the sorting model is completed, it is implemented in hardware on a heterogeneous multi-core processor for thread scheduling.

The implementation of the thread scheduling method for heterogeneous multi-core processors based on the Gale-Shapley algorithm will be described in detail below by taking the heterogeneous multi-core processor scheduling architecture shown in FIG. 2 as an example.

Suppose there are four threads T0, T1, T2, T3 running on the heterogeneous multi-core processor. During initialization, since there is no prior information of the thread, it is randomly scheduled on four cores, for example, the following matching methods (T0, core0), (T1, core1), (T2, core2), (T3, core3) are obtained.

After running for a period of time, each monitor collects the program dynamic features of its core, sends them to the corresponding T-ranker and C-ranker respectively, and obtains the following sorting results:

Table 1 The sorting results of threads to cores

Table 2 Sorting results of checking threads

After getting the above sorted list, send the sorting result to Matchmaker, and Matchmaker finds an optimal stable match according to the Gale-Shapley algorithm:

First, T0 makes a request to Core2 according to its selection priority. Core2 has no matching object at this time, and accepts T0's request to form a matching pair (T0, Core2);

Then, T1 makes a request to Core1 according to its selection priority. Core1 has no matching object at this time, and accepts T1's request to form a matching pair (T1, Core1);

Then, T2 makes a request to Core2 according to its selection priority. Core2 has already matched T0 at this time. Core2 checks its acceptance priority and finds that T2 has a higher priority than T0, so it accepts the request from T2 and forms a new match. pair(t2, core2);

Since Core2 is re-matched with T2, T0 loses the matching object, and it makes a request to Core1 in descending order. TO has a higher priority than T1 in the sorting list of Core1, so Core1 chooses to accept and form a new matching pair (T0, Core1) .

By analogy, the thread makes a matching request to the core from high to low according to its priority order. If the core has no matching object, it chooses to accept the request to form a matching pair with it; if the core already has a matching object, compare the new thread with the matching object. The priority of the object, if the priority of the new thread is higher than the previously accepted thread, then choose to accept the new thread as the matching object, if the priority of the new thread is lower than the previously accepted thread, reject the new request; rejected The thread reselects the next core in the sorting list to make a matching request; until all threads and cores have found matching objects, the matching process according to the Gale-Shapley algorithm ends. Theory proves that this matching must be in a stable state, and it is the best one among all stable matchings. And there is no doubt that the matching obtained according to the above process is Pareto optimal, because no thread (or core) can improve its own income without compromising the income of other threads (or cores).

The final stable matching is: (T0, Core1), (T1, Core3), (T2, Core2), (T3, Core0). The Scheduler schedules the threads to run on the corresponding cores according to the matching results. Monitor continues to collect new program features to prepare for the next dispatch.

The above is only an embodiment of the present invention, and there are many other situations that can be deduced in the same way, and are not listed one by one. In particular, the present invention only uses the ranking algorithm RankBoost to obtain the ranking results, and combines the Gale-Shapley algorithm to obtain stable matching for thread scheduling. You can also use other sorting algorithms combined with the Gale-Shapley algorithm to achieve the same matching results, such as AdaRank, Rank SVM, etc.

Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. These modifications and variations belong to the protection scope of the present invention.

Claims (9)

1. a heterogeneous polynuclear thread scheduling method is characterized in that, comprises that the behavioral characteristics according to program is respectively thread and karyogenesis sorted lists, and finds out the stable coupling of the optimum of thread and nuclear according to sorted lists, carries out thread scheduling according to this stable coupling.

2. heterogeneous polynuclear thread scheduling method as claimed in claim 1 is characterized in that, thread and karyogenesis sorted lists comprise the generation order models, specifically comprise the steps:

(1) selects an ideal data storehouse;

(2) extraction procedure sampling fragment from this database;

(3) the sampling of program fragment is moved at the simulator of each nuclear respectively, and obtained respective response, sampling of program fragment and response thereof are divided into training set and test set two parts;

(4) select suitable learning algorithm training order models;

(5) when the test error of order models meets the demands, the training stage finishes.

3. heterogeneous polynuclear thread scheduling method as claimed in claim 2 is characterized in that, this sampling of program fragment comprises proper vector, and for thread, this proper vector of a sampling of program fragment of input is exported the sorted lists to each nuclear; For nuclear, import this proper vector of each thread sampling of program fragment, be output as the sorted lists that each checks each thread.

4. heterogeneous polynuclear thread scheduling method as claimed in claim 1 is characterized in that, specifically comprises the steps:

Collect the operating all kinds of multidate informations of thread, be output as the proper vector of certain sampling of program fragment of thread;

Reception operates in the proper vector of the thread of this nuclear, and selects a prioritization for this thread to each nuclear according to it;

Checking each thread for each sorts;

Receive the sorted lists of each thread and nuclear, and find out the stable matching result of thread and nuclear;

Receive this matching result, dispatch by operating system, each thread is assigned on the corresponding nuclear moves.

5. as claim 1 or 4 described heterogeneous polynuclear thread scheduling methods, it is characterized in that this stable coupling of finding out thread and nuclear comprises the steps:

(1) thread proposes matching request to nuclear from high to low according to its prioritization, does not have match objects as fruit stone, and it is right with its formation coupling then to select to accept request;

(2) as fruit stone match objects has been arranged, then newer thread and the priority of match objects, the thread of accepting before if the priority of new thread is higher than, then select to accept new thread as match objects, if the thread of accepting before the priority of new thread is lower than is then refused new request;

(3) unaccepted thread is reselected next nuclear proposition matching request on the sorted lists, has all found match objects up to all threads and nuclear.

6. as claim 1 or 4 described heterogeneous polynuclear thread scheduling methods, it is characterized in that this stable coupling of finding out thread and nuclear comprises employing Gale-Shapley algorithm.

7. a heterogeneous polynuclear thread scheduling system is characterized in that, comprises information acquisition module, T sorting unit, C sorting unit, adaptation, thread scheduler, wherein:

Information acquisition module is used for collecting the operating all kinds of multidate informations of each thread, is output as the proper vector of certain sampling of program fragment of each thread;

The T sorting unit is used for receiving the proper vector that operates in the thread on this nuclear, and selects prioritization for this thread to each nuclear according to it;

The C sorting unit is used to each to check each thread and sorts;

Adaptation is used for receiving the sorted lists of each thread and each nuclear, and obtains the stable matching result of thread and nuclear;

Thread scheduler receives this matching result, dispatches by operating system, each thread is assigned on the corresponding nuclear moves.

8. heterogeneous multi-nucleus processor that adopts any one method of claim 1-6.

9. heterogeneous multi-nucleus processor that comprises claim 8.

Images