Berlin SPARQL Benchmark (BSBM) - Business Intelligence Use Case

Authors:
Chris Bizer (Web-based Systems Group, Freie Universität Berlin, Germany)
Andreas Schultz (Institut für Informatik, Freie Universität Berlin, Germany)
 
This version:
http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/20110217/BusinessIntelligenceUseCase/
Latest version:
http://www4.wiwiss.fu-berlin.de/bizer/BerlinSPARQLBenchmark/spec/BusinessIntelligenceUseCase/
Publication Date: 11/29/2010

Abstract

This document defines the Business Intelligence use case of the Berlin SPARQL Benchmark (BSBM) - Version 3 for measuring the performance of storage systems that expose SPARQL endpoints. The benchmark is built around an e-commerce use case, where a set of products is offered by different vendors and different consumers have posted reviews about products. This benchmark query mix is composed of queries that represent analytical questions by different stakeholders like vendors, customers or the owners of the e-commerce portal.

Table of Contents

1. Introduction

The SPARQL Query Language for RDF and the SPARQL Protocol for RDF are implemented by a growing number of storage systems and are used within enterprise and open web settings. As SPARQL is taken up by the community there is a growing need for benchmarks to compare the performance of storage systems that expose SPARQL endpoints via the SPARQL protocol. Such systems include native RDF stores, Named Graph stores, systems that map relational databases into RDF, and SPARQL wrappers around other kinds of data sources.

The Berlin SPARQL Benchmark (BSBM) defines a suite of benchmarks for comparing the performance of these systems across architectures. The benchmark is built around an e-commerce use case in which a set of products is offered by different vendors and consumers have posted reviews about products.  In this variation of the BSBM we focus on the SPARQL 1.1 Working Draft whose features are already implemented in various RDF stores. That is we consider grouping, aggregates and sub-queries.

The rest of this document is structured as follows: Section 2 defines the schema of benchmark dataset and describes the rules that are used by the data generator for populating the dataset according to the chosen scale factor. Section 3 defines the benchmark queries. Sections 4 defines how a system under test is verified against the qualification dataset.

2.Benchmark Dataset

All three scenarios use the same Benchmark Dataset .The bataset is built around an e-commerce use case, where a set of products is offered by different vendors and different consumers have posted reviews about products. The content and the production rules for the dataset are described in the BSBM Dataset Specification.

3. Benchmark Queries

This section defines a suite of benchmark queries and a query mix.

The benchmark queries are designed to emulate independent analytical queries over the dataset. These are:

  1. The top 10 most discussed product categories of products from a specific country based on number of reviews by reviewers from a certain country.
  2. The top 10 products most similar to a specific product, rated by the count of features they have in common.
  3. Products with largest increase of interest (ratio of review counts) from one month to the next.
  4. Feature with the highest ratio between price with that feature and price without that feature.
  5. Show the most popular products of a specific product type for each country - by review count.
  6. Reviewers who rated products by a specific Producer much higher than the average.
  7. Products in the top 1000 most offered products of a certain product type that are not sold by vendors of a specific country
  8. Find 10 cheapest vendors for a specific product type by the ratio of products below and above the average.

3.1 Query Mix

Query Mix

The complete query mix consists of 15 queries that simulate analytical questions posed by different stakeholder like vendors, producers, the portal owner and customers. The query sequence is given below:

  1. Query 1
  2. Query 3
  3. Query 2
  4. Query 2
  5. Query 4
  6. Query 5
  7. Query 5
  8. Query 6
  9. Query 5
  10. Query 6
  11. Query 6
  12. Query 7
  13. Query 8
  14. Query 7
  15. Query 6

3.2 SPARQL Queries for the Triple Data Model

Each query is defined by the following components:

Query 1: The top 10 most discussed product categories of products from a specific country based on number of reviews by reviewers from a certain country

Use Case Motivation: A vendor wants to find out which product categories get the most attention by people from a certain country

SPARQL Query:

prefix bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
prefix rev: <http://purl.org/stuff/rev#>

Select ?productType ?reviewCount
{
{ Select ?productType (count(?review) As ?reviewCount)
{
?productType a bsbm:ProductType .
?product a ?productType .
?product bsbm:producer ?producer .
?producer bsbm:country %Country1% .
?review bsbm:reviewFor ?product .
?review rev:reviewer ?reviewer .
?reviewer bsbm:country %Country2% .
}
Group By ?productType
}
}
Order By desc(?reviewCount) ?productType
Limit 10

Parameters:

Parameter Description
%Country1% A randomly selected Country URI.
%Country2% A randomly selected Country URI.

Query 2: The top 10 products most similar to a specific product, rated by the count of features they have in common.

Use Case Motivation:  A consumer wants to list similar products to the product they are viewing right now.

SPARQL Query:

  prefix bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/> 

SELECT ?otherProduct ?sameFeatures
{
?otherProduct a bsbm:Product .
FILTER(?otherProduct != %Product%)
{
SELECT ?otherProduct (count(?otherFeature) As ?sameFeatures)
{
%Product% bsbm:productFeature ?feature .
?otherProduct bsbm:productFeature ?otherFeature .
FILTER(?feature=?otherFeature)
}
Group By ?otherProduct
}
}
Order By desc(?sameFeatures) ?otherProduct
Limit 10

Parameters:

Parameter Description
%Product% A randomly selected Product URI.

Query 3: Products with largest increase of interest (ratio of review counts) from one month to the next.

Use Case Motivation: Some stakeholder wants to get a list of products with the highest increase in popularity in a certain period for further investigation.

SPARQL Query:

  prefix bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
prefix bsbm-inst: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/>
prefix rev: <http://purl.org/stuff/rev#>
prefix dc: <http://purl.org/dc/elements/1.1/>
prefix xsd: <http://www.w3.org/2001/XMLSchema#>

Select ?product (xsd:float(?monthCount)/?monthBeforeCount As ?ratio)
{
{ Select ?product (count(?review) As ?monthCount)
{
?review bsbm:reviewFor ?product .
?review dc:date ?date .
Filter(?date >= "%ConsecutiveMonth_1%"^^<http://www.w3.org/2001/XMLSchema#date> && ?date < "%ConsecutiveMonth_2%"^^<http://www.w3.org/2001/XMLSchema#date>)
}
Group By ?product
} {
Select ?product (count(?review) As ?monthBeforeCount)
{
?review bsbm:reviewFor ?product .
?review dc:date ?date .
Filter(?date >= "%ConsecutiveMonth_0%"^^<http://www.w3.org/2001/XMLSchema#date> && ?date < "%ConsecutiveMonth_1%"^^<http://www.w3.org/2001/XMLSchema#date>) #
}
Group By ?product
Having (count(?review)>0)
}
}
Order By desc(xsd:float(?monthCount) / ?monthBeforeCount) ?product
Limit 10

Parameters:

Parameter Description
%ConsecutiveMonth_X% The date of the first day of a randomly selected month (Index 0). With index X the date will be set X months after the picked dat

Query 4: Feature with the highest ratio between price with that feature and price without that feature.

Use Case Motivation: A customer wants to inform herself which features have the most impact on the product price to get hints on how to restrict the following product search.

SPARQL Query:

  prefix bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
prefix bsbm-inst: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/>
prefix xsd: <http://www.w3.org/2001/XMLSchema#>

Select ?feature (?withFeaturePrice/?withoutFeaturePrice As ?priceRatio)
{
{ Select ?feature (avg(xsd:float(xsd:string(?price))) As ?withFeaturePrice)
{
?product a %ProductType% ;
bsbm:productFeature ?feature .
?offer bsbm:product ?product ;
bsbm:price ?price .
}
Group By ?feature
}
{ Select ?feature (avg(xsd:float(xsd:string(?price))) As ?withoutFeaturePrice)
{
{ Select distinct ?feature {
?p a %ProductType% ;
bsbm:productFeature ?feature .
} }
?product a %ProductType% .
?offer bsbm:product ?product ;
bsbm:price ?price .
FILTER NOT EXISTS { ?product bsbm:productFeature ?feature }
}
Group By ?feature
}
}
Order By desc(?withFeaturePrice/?withoutFeaturePrice) ?feature
Limit 10

Parameters:

Parameter Description
%ProductType% A randomly selected Class URI from the class hierarchy (except root category).

Query 5: Show the most popular products of a specific product type for each country - by review count

Use Case Motivation: For advertisement reasons the owners of the e-commerce platform want to generate profiles for the two dimensions product type and the country of a customer.

SPARQL Query:

  prefix bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
prefix bsbm-inst: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/>
prefix rev: <http://purl.org/stuff/rev#>
prefix xsd: <http://www.w3.org/2001/XMLSchema#>

Select ?country ?product ?nrOfReviews ?avgPrice
{
{ Select ?country (max(?nrOfReviews) As ?maxReviews)
{
{ Select ?country ?product (count(?review) As ?nrOfReviews)
{
?product a %ProductType% .
?review bsbm:reviewFor ?product ;
rev:reviewer ?reviewer .
?reviewer bsbm:country ?country .
}
Group By ?country ?product
}
}
Group By ?country
}
{ Select ?country ?product (avg(xsd:float(xsd:string(?price))) As ?avgPrice)
{
?product a %ProductType% .
?offer bsbm:product ?product .
?offer bsbm:price ?price .
}
Group By ?country ?product
}
{ Select ?country ?product (count(?review) As ?nrOfReviews)
{
?product a %ProductType% .
?review bsbm:reviewFor ?product .
?review rev:reviewer ?reviewer .
?reviewer bsbm:country ?country .
}
Group By ?country ?product
}
FILTER(?nrOfReviews=?maxReviews)
}
Order By desc(?nrOfReviews) ?country ?product

Parameters:

Parameter Description
%ProductType% A randomly selected Class URI from the class hierarchy (except root category).

Query 6: Reviewers who rated products by a specific Producer much higher than other reviewer.

Use Case Motivation: The stakeholders representing the e-commerce platform want to find potential spam reviewer, who rate products by a specific producer much higher than the average.

SPARQL Query:

  prefix bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
prefix bsbm-inst: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/>
prefix rev: <http://purl.org/stuff/rev#>

Select ?reviewer (avg(xsd:float(?score)) As ?reviewerAvgScore)
{
{ Select (avg(xsd:float(?score)) As ?avgScore)
{
?product bsbm:producer %Producer% .
?review bsbm:reviewFor ?product .
{ ?review bsbm:rating1 ?score . } UNION
{ ?review bsbm:rating2 ?score . } UNION
{ ?review bsbm:rating3 ?score . } UNION
{ ?review bsbm:rating4 ?score . }
}
}
?product bsbm:producer %Producer% .
?review bsbm:reviewFor ?product .
?review rev:reviewer ?reviewer .
{ ?review bsbm:rating1 ?score . } UNION
{ ?review bsbm:rating2 ?score . } UNION
{ ?review bsbm:rating3 ?score . } UNION
{ ?review bsbm:rating4 ?score . }
}
Group By ?reviewer
Having (avg(xsd:float(?score)) > min(?avgScore) * 1.5)

Parameters:

Parameter Description
%Producer% A producer URI (randomly selected)

Query 7: Products in the top 1000 most offered products of a certain product type that are not sold by vendors of a specific country

Use Case Motivation: A vendor wants information about potential market niches to offer new products in the vendor's country of origin.

SPARQL Query:

  prefix bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
prefix bsbm-inst: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/>
prefix xsd: <http://www.w3.org/2001/XMLSchema#>

Select ?product
{
{ Select ?product
{
{ Select ?product (count(?offer) As ?offerCount)
{
?product a %ProductType% .
?offer bsbm:product ?product .
}
Group By ?product
}
}
Order By desc(?offerCount)
Limit 1000
}
FILTER NOT EXISTS
{
?offer bsbm:product ?product .
?offer bsbm:vendor ?vendor .
?vendor bsbm:country ?country .
FILTER(?country=%Country%)
}
}

Parameters:

Parameter Description
%Country% A country URI (randomly selected)
%ProductType% A random product type (all levels)

Query 8: Find 10 cheapest vendors for a specific product type by the highest percentage of their products below the average.

Use Case Motivation: A vendor or customer wants to find "discounter" vendors for competitor analyses and procurement respectively.

SPARQL Query:

  prefix bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
prefix bsbm-inst: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/>
prefix xsd: <http://www.w3.org/2001/XMLSchema#>

Select ?vendor (xsd:float(?belowAvg)/?offerCount As ?cheapExpensiveRatio)
{
{ Select ?vendor (count(?offer) As ?belowAvg)
{
{ ?product a %ProductType% .
?offer bsbm:product ?product .
?offer bsbm:vendor ?vendor .
?offer bsbm:price ?price .
{ Select ?product (avg(xsd:float(xsd:string(?price))) As ?avgPrice)
{
?product a %ProductType% .
?offer bsbm:product ?product .
?offer bsbm:vendor ?vendor .
?offer bsbm:price ?price .
}
Group By ?product
}
} .
FILTER (xsd:float(xsd:string(?price)) < ?avgPrice)
}
Group By ?vendor
}
{ Select ?vendor (count(?offer) As ?offerCount)
{
?product a %ProductType% .
?offer bsbm:product ?product .
?offer bsbm:vendor ?vendor .
}
Group By ?vendor
}
}
Order by desc(xsd:float(?belowAvg)/?offerCount) ?vendor
limit 10

Parameters:

Parameter Description
%ProductType% A randomly selected Class URI from the class hierarchy (all levels).

3.4 SQL Queries for the Relational Data Model

This section will contain a SQL representation of the benchmark queries in order to be able to compare the performance of stores that expose SPARQL endpoints to the performance of classic SQL-based RDBMS.
TODO: Translate SPARQL queries into SQL once they are stable.

4. Qualification Dataset and Tests

TODO: Adapt to Use Case
Before the performance of a SUT is measured, it has to be verified that the SUT returns correct results for the benchmark queries.

For testing whether a SUT returns correct results, the BSBM benchmark provides a qualification dataset and a qualification tool which compares the query results of a SUT with the correct query results. At the moment, the qualification tool verifies only the results of SELECT queries. The results of DESCRIBE and CONSTRICT queries (queries 9 and 12) are not checked.

A BSBM qualification test is conducted in the two-step procedure described below:

  1. Load the qualification dataset into the SUT, fetch the Test Driver data for the qualification dataset and unzip the "td_data" directory into the root directory of the BSBM tools, then run a qualification test against the SUT. The qualification test is supported by the Test Driver by running it with the -q parameter.  Example:
    $ java -cp bin:lib/* benchmark.testdriver.TestDriver -q http://SUT/sparql

    where http://SUT/sparql specifies the SPARQL endpoint
    This will create a qualification file named "run.qual" (different file name can be specified with the "-qf" parameter) which is used in step 2. Also the run.log (if logging is set to "ALL" in the log4j.xml file) contains all queries with full result text, so single queries can be examined later on.
  2. Use the Qualification tool from the benchmark.qualification package. It has the following options:
Option Description
-rc Only check the amount of results returned and not the result content.
-ql <qualification log file name> Specify the file name to write the qualification test results into.

You also need a correct qualification file to test your own qualification file against. Then run the Qualification test. Example:
$ java -cp bin:lib/* benchmark.qualification.Qualification correct.qual run.qual

where run.qual is the qualification file generated by the Test Driver in qualification mode
This generates by default a log file called "qual.log" with the following content: A variation does not always mean an error. For example if the SUT returns an  xsd:dateTime value in another (correct) format than expected. So variations should always be inspected in the verbose log file which was generated by the Test Driver (run.log). For comparison means a Test Driver log file is available of a correct run.

5. References

For more information about RDF and SPARQL Benchmarks please refer to:

ESW Wiki Page about RDF Benchmarks

Other SPARQLBenchmarks

Papers about RDF and SPARQL Benchmarks

Appendix A: Changes

Appendix B: Acknowledgements

The work on the BSBM Benchmark Version 3 is funded through the LOD2 project.