image_pdfimage_print

Here we show how to stress test a Cassandra cluster using the cassandra-stress tool.

What this tool does is run inserts and queries against a table that it generates by itself or an existing table or tables.

The Basic Flow

The test works like this:

  • Insert random data. You specify the length of text fields or random numeric values by selecting a fixed value or a statistical distribution such as a normal, uniform, or other distribution. The normal distribution are values drawn from some mean and standard deviation, i.e. the familiar bell curve. The uniform distribution is random numbers drawn from a range like 1,2,3,….
  • Run select statements using the values generated.
  • Calculate the time in milliseconds to run each operation.
  • Calculate the mean time, standard deviations number of garbage collections etc. for each iteration. This gives you an average and the bell curve so you can see how widely your operations are disbursed. It gives you graphs over time so you can see whether performance degrades over time. Lots of operations far from the mean indicate a high level of variance. This could point to items you need to tune, such as indexes, partitions, add more memory, etc.

Test Setup

You need a Cassandra cluster. If you do not have one yet follow these instructions.

We need to create a keyspace, table, and index and to create a stress test configuration file in YAML format.

Now deactivate Python 2.7 virtual environment, if you are using that, then run cqlsh. Paste in the following Cassandra SQL.

CREATE KEYSPACE Library
WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 };

CREATE TABLE Library.book (
ISBN text,
copy int,
title text,
PRIMARY KEY (ISBN, copy)
);

create index on library.book (title);

Now, create a text file named stress-books.yaml and paste the following into it. This is a YAML format, so don’t mess up the indentation.

Below we explain the fields.

keyspace: library

table: book

columnspec:
  - name: text
    size: uniform(5..10) 
    population: uniform(1..10) 
  - name: copy
    cluster: uniform(20..500) 
  - name: title
    size: uniform(5..10)

insert:
  partitions: fixed(1) 
  select: fixed(1)/500 
  batchtype: UNLOGGED         

queries:
   books:
      cql: select *  from book where title = ?
      fields: samerow

The field names are:

keyspace Name of existing keyspace. You could also put SQL here to create one if it does not exist.
table Name of existing table. You could also put SQL here to create one if it does not exist.
insert, queries These are the functions we will call. The insert does the insert and the queries/books does the query we put there. It runs queries against the values inserted in the batch it just ran.
columnspec:
– name: text
size: uniform(5..10)
There is one columnspec for each value we want to populate. The size is the length of the field. uniform(5..10) means to generate a text string from 5 to10 characters. If it was a numeric field it would create random numbers in that range.
insert:
partitions: fixed(1)
select: fixed(1)/500
batchtype: UNLOGGED
This means to insert a fixed number of rows in each partition in each batch. We explained batch operations here.

Run the Stress Test

Run the command below to start the test. The arguments are:

cassandra-stress user profile=stress-books.yaml ops\(insert=1,books=1\) n=10000  -graph file=stress.html
user profile=stress-books.yaml means use stress-books.yaml for instructions.
n=10000
ops\(insert=1,books=1\)
Here we say run 10,000 batches and for each run the insert code 1 time and the books code 1 time. If you leave this off it will run until the standard deviation of the error is < 0.2.
-graph file=stress.html This will create an html file called stress.html that will graph the results of different metrics.

The output is quite long. Here is part of it truncated.

******************** Stress Settings ********************
Command:
Type: user
Count: 10,000


…

Connected to cluster: Walker Cluster, max pending requests per connection 128, max connections per host 8
Datatacenter: datacenter1; Host: localhost/127.0.0.1; Rack: rack1
Generating batches with [1..1] partitions and [0..1] rows (of [20..500] total rows in the partitions)

Sleeping 2s...
Warming up insert with 500 iterations...
Warming up books with 500 iterations...
Thread count was not specified

It will echo metrics for each batch iteration. You can copy and paste this test and then import it into Excel or Google Sheets to make that easier to read.

type       total ops,    op/s,    pk/s,   row/s,    mean,     med,     .95,     .99,    .999,     max,   time,   stderr, errors,  gc: #,  max ms,  sum ms,  sdv ms,      mb
books,          1003,    1003,     383,     387,     5.0,     3.9,    12.5,    20.2,    29.7,    40.7,    1.0,  0.00000,      0,      2,      10,      18,       1,     319
insert,          825,     825,     825,     934,     2.7,     2.0,     7.3,    12.1,    16.9,    23.8,    1.0,  0.00000,      0,      2,      10,      18,       1,     319
total,          1828,    1828,    1208,    1321,     4.0,     2.9,    11.1,    18.1,    27.2,    40.7,    1.0,  0.00000,      0,      2,      10,      18,       1,     319
books,          1096,     930,     350,     360,     5.8,     3.9,    14.1,    27.4,    33.3,    33.3,    1.1,  0.11339,      0,      0,       0,       0,       0,       0
insert,          904,     790,     790,     910,     3.0,     2.3,     9.0,    12.1,    12.3,    12.3,    1.1,  0.11339,      0,      0,       0,       0,       0,       0
total,          2000,    1720,    1140,    1270,     4.5,     3.2,    11.9,    21.9,    33.3,    33.3,    1.1,  0.11339,      0,      0,       0,       0,       0,       0

Do that and then the spreadsheet is easier to read.

This shows the total numbers of inserts and selects (books is what we named that) and the mean (average) number of milliseconds required to do that. In this example I ran earlier it ran 2,000 times.

A statistician looking at this would want to know 2, 3, and 4 standard deviations from the mean. That corresponds to 95%, 99%, and 99.9% of the transactions. In other words those are far from the mean.

Below is the right-hand side of the spreadsheet. Not all the metrics can fit onto this screen, such as number of garbage collections (the number of times that the Java Virtual Machine had to clear its cache. When it does that all operations pause.) But this does not matter as the HTML file will give the complete picture.

Open the file stress.html with a browser.

You can see that the test took 22 seconds running on my 2 cluster t2.large Amazon Ubuntu machines. The spikes in operation time coincided with garbage collections. You can see that by selecting the gc# option in the graph type drop down.

The stats show that the average throughout was 2,231 inserts per second.

You can pick from the drop down at the top right of the chart to plot different metrics.

 

Wikibon: Automate your Big Data pipeline

Learn how data management experts throughout the industry are transforming their Big Data infrastructure for maximum business impact.
Download Now ›
Last updated: 12/20/2018

These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.

See an error or have a suggestion? Please let us know by emailing blogs@bmc.com.

About the author

Walker Rowe

Walker Rowe

Walker Rowe is an American freelance tech writer and programmer living in Tunisia. He specializes in big data, analytics, and programming languages. Find him on LinkedIn or Upwork.