This document answers frequently-asked questions about WebStone.
If you have comments about this document, please forward them to webstone@mindcraft.com.
This is a list of answers to Frequently Asked Questions (FAQ) about WebStone. The latest copy is always available at http://www.mindcraft.com/WebStone/ and via the WebStone mailing list. The FAQ is periodically posted to the WebStone mailing list, and to the USENET newsgroup comp.benchmarks.
WebStone is a benchmark program that measures the performance of a Web server by running processes that send HTTP protocol requests to the Web server. WebStone is a client-server program that allows one or more computers to cooperate to provide the load to test a server.
The original WebStone benchmark was released by SGI in March, 1995. The original white paper describing this benchmark is available from http://www.mindcraft.com/WebStone/. It is also included in the WebStone distribution, as WebStone/doc/webstone-white-paper.ps.
WebStone is currently maintained by Mindcraft, Inc.
WebStone is not a proprietary benchmark, it is an open benchmark. The source code is freely available, and anyone can examine it. By design, WebStone does not favor any one company. It is simply a performance measurement tool.
WebStone 2.0 is a rewrite of the WebStone 1.1 code. Significant changes were made both in the code and in the fileset and run rules. Many bugs were eliminated, support for other platforms has been included and many new features have been added. The WebStone 1.1 and WebStone 2.0 numbers cannot be compared, since so much has changed. In general, WebStone 1.1 will give higher connections/second values, but lower throughput numbers than WebStone 2.0.
WebStone 2.0 includes support for testing proxy servers, as well as more flexible handling of URLs that enables WebStone to test a wide variety of content types. The code has also been significantly rewritten so that it is more robust and portable
Webstone 2.5 provides many more bug fixes and portability enhancements, but does not change the performance characteristics of the benchmark.
Absolutely NOT! WebStone 1.1 numbers are based on a different fileset, as well as an older version of the benchmarking software. The WebStone 1.1 fileset was based on a fileset with a smaller average filesize, so that the number of connections per second will tend to be higher (all things being equal). The WebStone 2.0 fileset is based on observations of several real world sites, and the distribution of the filesizes found there. This fileset is also similar to the fileset chosen by the SPEC committee for their benchmark.
While it is possible to convert the 1.1 fileset to a 2.0 format and then test it, the resulting numbers will not be the same, because the underlying software used to perform the testing has changed. WebStone 1.1 was also heavily abused because of the lack of run rules, and reporting rules. It is recommended that everyone move to WebStone 2.0 or to WebStone 2.5.
Yes. The WebStone 2.5 release leaves the basic functionality of WebStone 2.0 unchanged. WebStone 2.5 results are comparable to WebStone 2.0 results if the same fileset and the same load module (for CGI and API tests) are used.
WebStone 2.5 reports throughput numbers that are 4.9% larger than those reported by WebStone 2.0. This is because the benchmark has been corrected to use the standard definition of Mbps, which is 1,000,000 bits per second rather than 220 bits per second.
A SPEC working group has published a Web server benchmarking program, SPECweb96. Mindcraft uses the SpecWeb96 benchmark and we support fully the effort. WebStone is easier than SPECweb96 to adapt to the user's particular needs. SPECweb96 does not replace WebStone.
Basically, if you like WebStone, use it. If you want SPECweb96 numbers, buy the benchmark from SPEC and check it out.
WebStone makes HTTP 1.0 GET requests for pages on a Web server. The user specifies which URLs are to be requested and the relative frequency of requests for the different URLs. Any Web server can be tested, and any HTML content can be used.
WebStone measures the throughput and latency of each HTTP transfer. By default, only statistical data are returned, but the user may optionally request data for each and every transaction. WebStone also reports transaction failures, which translate into those little "Connection Refused" alerts in the real world.
WebStone does not yet do any of the following (listed roughly in order of planned implementation):
If you have additional requests for WebStone functionality, contact the WebStone mailing list.
WebStone creates a load on a Web server by running one or more webclient processes on one or more client computers. Each webclient process makes the same sorts of requests that a Web server would receive from a Web browsers or from other Web data retrieval software. Each webclient process makes requests as fast as it can receive data back from the server being tested, so it creates a larger load than a single interactive user would. It is not uncommon to run 100 or more webclient processes on a single computer.
The testing done by the webclient processes is managed by a single process called webmaster. It can be run on one of the client computers or on a separate computer. Webmaster distributes the Web client software and test configuration files to the client computers. Webmaster then starts a benchmark run and waits for the webclients to report back the performance they measured. The Webmaster combines the performance results from all the Web clients into a single summary report.
The figure below shows the relationship between the various parts of a Webstone test lab:
For more details, see the manual pages and the README files in the WebStone 2.5 distribution.
WebStone's job is to send requests to a server. Each webclient process sends requests as fast as it receives data back from the server. The user chooses how many processes to run and chooses the type of data that the processes request.
The standard run rules for WebStone 2.0.1 and WebStone 2.5 require the use of a standard file set to allow easy comparison between performance results for different systems. This standard file set is small enough that most Web servers can cache all the testdata. This means that the standard WebStone benchmark tests give an indication of the speed of Web server and operating system software, CPU speed, and network speed. With the standard run rules WebStone is not very sensitive to the speed of disk access.
The test engineer is free to choose a different file set if the goal is to simulate a particular Web site or a specific type of server access. This makes WebStone a powerful and flexible tool for Web site modelling.
WebStone tests Web servers using three different types of access methods:
WebStone comes with API tests for Netscape's API (NSAPI) and for for Microsoft's Internet Information Server API (ISAPI). In order to do the testing for a different API, the tester must create a load-generating module that is comparable to the NSAPI and ISAPI modules.
The WebStone 2.0.1 CGI and API tests do not access a file directly. Instead they generate random data which they return in response to a request. These tests are compute intensive and may be more sensitive to CPU speed than to the efficiency of the CGI and API interfaces in the Web server software.
For WebStone 2.5 new CGI and API modules were created that are more sensitive to the efficiency of the access method being tested. These modules make a few CGI or API calls and return a file.
The latest versions of WebStone and of this FAQ are available at http://www.mindcraft.com/WebStone .
Here's a Web page with a brief overview.
WebStone primarily measures throughput (bytes/second) and latency (time to complete a request). WebStone also reports pages/minute, connection rate averages, and other numbers. Some of these may help you to sanity-check the throughput measurements.
Two types of throughput are measured: aggregate and per-client. Aggregate throughput is simply total bytes (body + header) transferred throughout the test, divided by the total test time. Per-client throughput divides total bytes by the total connect time and then by the number of clients.
Two types of latency are reported: connection latency and request latency. For each metric, the mean time is provided, as well as the standard deviation of all data, plus the minimum and maximum times. Connection latency reflects the time taken to establish a connection, while request latency reflects the time to complete the data transfer once the connection has been established.
User-perceived latency will include the sum of connection and request latencies, plus any network latency due to WAN connections, routers, modems, etc.
WebStone also reports a metric called Little's Ls. Ls is derived from Little's Law, and reflects how much time is spent by the server on request processing, rather than overhead and errors. Ls. is also an indirect indicator of the average number of connections which the web server has open at any particular instant. This number should stay very close to the number of clients, or else some clients are being denied access to the server at any given time.
If you load your Web servers high enough, you'll begin to see errors in the results. That's fine, as far as WebStone is concerned. It means that your server is heavily loaded, and some client rtequests aren't being serviced before they time out. The number of errors at a given load can be an excellent indicator of how your server will perform under extremely heavy loads.
Subscribe to the WebStone mailing list! Send a message to majordomo@webstone.mindcraft.com - the subject doesn't matter, but the content should be:
subscribe webstone
You should receive a message shortly, confirming that you've been added to the mailing list. You can send to the whole list at webstone@mindcraft.com - the authors of WebStone read the list, and they'll do their best to help. Other list members may also be able to help.
If you have access to USENET News, you can also read and post to comp.benchmarks. As with any newsgroup, read the FAQ before posting!
There's also a mailing list devoted to the performance limits of the HTTP protocol. You can subscribe by sending e-mail to www-speed-request@tipper.oit.unc.edu with the text
subscribe <your-email-address>
This file and all files contained in the WebStone distribution are copyright © 1998, Mindcraft, Inc.
This software is provided without support and without any obligation on the part of Mindcraft, Inc. to assist in its use, correction, modification or enhancement. There is no guarantee that this software will be included in future software releases, and it probably will not be included.
THIS SOFTWARE IS PROVIDED "AS IS" WITH NO WARRANTIES OF ANY KIND INCLUDING THE WARRANTIES OF DESIGN, MERCHANTIBILITY AND FITNESS FOR A PARTICULAR PURPOSE, OR ARISING FROM A COURSE OF DEALING, USAGE OR TRADE PRACTICE.
In no event will Mindcraft, Inc. be liable for any lost revenue or profits or other special, indirect and consequential damages, even if Mindcraft, Inc. has been advised of the possibility of such damages.
