TU Delft
The GrenchMark Framework
Parallel and Distributed Systems
EWI PDS GrenchMarkThe GrenchMark Framework
Quick links
about process extend typical use
    implementation references

About the GrenchMark framework
what is GrenchMark?


GrenchMark is a framework for synthetic grid workload generation and submission, which has been designed, implemented, and deployed by the MultiProbe team in the Parallel and Distributed Systems group of the Faculty of Electrical Engineering, Mathematics, and Computer Science of the Delft University of Technology. GrenchMark is extensible, in that it allows new types of grid applications to be included in the workload generation, parameterizable, as it allows the user to parameterize the workloads generation and submission, and portable, as its reference implementation is written in Python.

The workload generator is based on the concepts of unit generators and of job description files (JDF) printers. The unit generators produce detailed descriptions on running a set of applications (workload unit), according to the workload description provided by the user. In principle, there is one unit for each supported application type. The printers take the generated workload units and create job description files suitable for grid submission. In this way, multiple unit generators can be coupled to produce a workload that can be submitted to any grid resource manager, as long as the resource manager supports that type of applications.

The grid applications currently supported by GrenchMark are sequential jobs, jobs which use MPI, and Ibis jobs. GrenchMark includes a set of over 35 synthetic and real applications. We have implemented four synthetic applications: sser, a sequential application with parameterizable computation and memory requirements, sserio, a sequential application with parameterizable computation and I/O requirements, smpi1, an MPI application with parameterizable computation, communication, memory, and I/O requirements, and swf, an application implementing the job model used in the Standard Workloads Format, from the Parallel Workloads Archives (PWA). We use all the real applications and some of the synthetic applications included in the default Ibis distribution package. The reason is threefold: the Ibis applications closely resemble or are real-life parallel applications, they have been proven to run on a variety of grid settings, and they have been thoroughly described. For the complete list of publications related to Ibis applications, please visit http://www.cs.vu.nl/ibis. The Ibis applications come from the areas of physical simulations, parallel rendering, computational mathematics, state space search, bioinformatics, data compression, grid methods, and optimization. Currently, GrenchMark can submit jobs to Koala}, Globus GRAM, and Condor.

The workload submitter generates detailed reports of the submission process. The reports include all job submission commands, the turnaround time of each job, including the grid overhead, the total turnaround time of the workload, and various statistical information.

Replaying traces with GrenchMark

GrenchMark can replay traces from various production environments. First, the user can load traces in the Standard Workload Format. Second, since a real trace contains tens of thousands of jobs, filtering out jobs and selecting the first-N jobs according to various criteria is essential for selecting a usable workload; GrenchMark can be used to shape and fit a real workload trace. Third, real traces cannot usually be replayed on another system than the one on which they were acquired, and even on the same system if it has changed. The latter happens more often in the case of grids, which are naturally evolving with their middleware, and therefore are very dynamic. GrenchMark can scale various aspects of the workload, e.g., the requested resources, or the job runtimes, and can help alleviate this problem.

Modeling workloads with GrenchMark

GrenchMark offers support for the following workload modeling aspects. First, it supports unitary and composite applications, and single-site and co-allocated jobs. Second, it allows the user to define various job inter-arrival times based on well-known statistical distributions. Besides the Poisson distribution, used traditionally in queue-based systems simulation, GrenchMark also supports uniform, normal, exponential and hyper-exponential, Weibull, log normal, and gamma distributions. Third, it allows the workload designer to combine several workloads into a single one (mixing). This allows for instance the inclusion of bursts, by combining a short workload with many jobs per time unit with a longer one, comprising fewer jobs per time unit. An additional use of workload mixing is in a what-if analysis that evaluates what will happen to a grid community if its resources would be shared with another group of users. In this case, the workload modeler can mix the typical workload of the two communities and evaluate whether the system can support both, under various job acceptance and execution policies.

The GrenchMark process
Figure MP-1. The GrenchMark process.

The GrenchMark process
brief description

We define two use cases for the GrenchMark framework: in real world and in simulations. A workload generated by GrenchMark using only synthetic and well studied applications can be equally used in both cases.

Figure Figure MP-1 depicts the typical process of using the GrenchMark framework in a real environment. First, the user describes the workload to be generated, as a formatted text file (1). Based on the user description, on the known application types, and on information about the grid sites, a workload is then generated by GrenchMark (2), and submitted to the grid (3). The grid environment is responsible for executing the jobs and returning their results (4). The results include not only job outcomes, but also detailed submission reports. Finally, the user processes all results in a post-production step (5). The use of GrenchMark for simulations is similar, with steps (3) and (4) possibly combined.


Extending the GrenchMark framework
brief description

GrenchMark has been designed with an incremental approach in mind, and facilitates future extensions. The framework design can be easily extended, for instance by adding various workload generation notions (e.g., users, virtual organizations). GrenchMark also offers a simple plug-in system, which can be used to add unit generators (new application types) and printers (support for other grid resource managers).

Another way the user can extend the workload generation process is to define a more refined workload specification language and use it from an already existing plug-in; the only requirement is that the extension language is based on Key=Value statements. A file written in an extension language is automatically parsed, and the data is available to the user when the plug-in is invoked. GrenchMark further offers mechanisms to parse simple values (e.g., booleans, strings, integers, and floats), and more complicated constructs (e.g., lists, and lists with weights).


Typical use
brief description

The GrenchMark framework can be used in a wide-range of situations; we list below a number of them:

  1. Grid system analysis
    1. Performance testing and analysis
    2. What-if analysis
  2. Functionality testing in grid environments
    1. System functionality testing
    2. Periodic system testing
  3. Comparing grid settings
    1. Performance comparison of single site vs. co-allocated jobs
    2. Performance comparison of unitary vs. composite jobs
    3. Jobs throughput comparison of two grid systems


brief description
We have implemented the GrenchMark framework in
Python [1].


we refer to these links for detailed information
[1] Python official web site. http://www.python.org.


Last modified: Tue, 29 November, 2005 7:11 AM
The newest version of this page can be found at: http://grenchmark.st.ewi.tudelft.nl/grenchmark.html
Copyright © 1998-2005 Alexandru Iosup. All Rights Reserved.
And the famous NedStat counter: