This document defines the Berlin SPARQL Benchmark (BSBM) for measuring the performance of storage systems that expose SPARQL endpoints. Such systems include native RDF stores, Named Graph stores, systems that map relational databases into RDF, and SPARQL wrappers around other kinds of data sources. The benchmark is built around an e-commerce use case, where a set of products is offered by different vendors and different consumers have posted reviews about products. The benchmark query mix illustrates the search and navigation pattern of a consumer looking for a product.
The SPARQL Query Language for RDF and the SPARQL Protocol for RDF are implemented by a growing number of storage systems and are used within enterprise and open web settings. As SPARQL is taken up by the community there is a growing need for benchmarks to compare the performance of storage systems that expose SPARQL endpoints via the SPARQL protocol. Such systems include native RDF stores, Named Graph stores, systems that map relational databases into RDF, and SPARQL wrappers around other kinds of data sources.
The Berlin SPARQL Benchmark (BSBM) defines a suite of benchmarks for comparing the performance of these systems across architectures. The benchmark is built around an e-commerce use case in which a set of products is offered by different vendors and consumers have posted reviews about products. The benchmark query mix illustrates the search and navigation pattern of a consumer looking for a product.
The Berlin SPARQL Benchmark (BSBM) consists of:
The Berlin SPARQL Benchmark was designed along three goals: First, the benchmark should allow the comparison of different storage systems that expose SPARQL endpoints across architectures. Testing storage systems with realistic workloads of use case motivated queries is a well established benchmarking technique in the database field and is for instance implemented by the TPC benchmarks. The Berlin SPARQL Benchmark should apply this technique to systems that expose SPARQL endpoints. As an increasing number of Semantic Web applications do not rely on heavyweight reasoning but focus on the integration and visualization of large amounts of data from autonomous data sources on the Web, the Berlin SPARQL Benchmark should not be designed to require complex reasoning but to measure the performance of queries against large amounts of RDF data.
This document defines version 2 of the BSBM Benchmark. Compared to version 1, which has been released in September 2008, version 2 contains two additional benchmark queries, a relational representation of the benchmark dataset and a SQL representation of the benchmark queries. In addition, the data generation rules and the benchmark queries were fine-tuned based on the experience from running version 1 against various stores.
The rest of this document is structured as follows: Section 2 defines the schema of benchmark dataset and describes the rules that are used by the data generator for populating the dataset according to the chosen scale factor. Section 3 defines the benchmark queries. Section 4 describes the performance metrics that are calculated by the test driver. Section 5 defines the rules that must be met when running the benchmark. Section 6 describes different benchmarking scenarios. Section 7 specifies the content of the BSBM full-disclosure report. Sections 8 and 9 define how a system under test is verified against the qualification dataset and describe the usage of the data generator and test driver.
This section defines the logical schema of the BSBM benchmark dataset (2.1) and the RDF triple, Named Graphs and relational representation of this schema. Section 2.3 defines the data generation rules that are used by the data generator to populate the dataset according to a given scale factor.
This section defines the logical schema for the benchmark dataset. The dataset is based on an e-commerce use case, where a set of products is offered by different vendors and different consumers have posted reviews about these products on various review sites.
| Prefix | Namespace |
| rdf: | http://www.w3.org/1999/02/22-rdf-syntax-ns# |
| rdfs: | http://www.w3.org/2000/01/rdf-schema# |
| foaf: | http://xmlns.com/foaf/0.1/ |
| dc: | http://purl.org/dc/elements/1.1/ |
| xsd: | http://www.w3.org/2001/XMLSchema# |
| rev: | http://purl.org/stuff/rev# |
| bsbm: | http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/ |
| bsbm-inst: | http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/ |
Meta data Properties
The meta data properties are used to capture the information source and the publication date of each instance.
Class Product
Comment: Products are described by different sets of product properties and product features.
Example RDF Instance:
dataFromProducer001411:Product00001435443
rdf:type bsbm:Product;
rdf:type bsbm-inst:ProductType001342;
rdfs:label "Canon Ixus 20010" ;
rdfs:comment "Mit ihrer hochwertigen Verarbeitung, innovativen Technologie und faszinierenden Erscheinung
verkörpern Digital IXUS Modelle die hohe Kunst des Canon Design." ;
bsbm:producer bsbm-inst:Producer001411 ;
bsbm:productFeature bsbm-inst:ProductFeature003432 ;
bsbm:productFeature bsbm-inst:ProductFeature103433 ;
bsbm:productFeature bsbm-inst:ProductFeature990433 ;
bsbm:productPropertyTextual1 "New this year." ;
bsbm:productPropertyTextual2 "Special Lens with special focus." ;
bsbm:productPropertyNumeric1 "1820"^^xsd:Integer ;
bsbm:productPropertyNumeric2 "140"^^xsd:Integer ;
bsbm:productPropertyNumeric3 "17"^^xsd:Integer ;
dc:publisher dataFromProducer001411:Producer001411 ;
dc:date "2008-02-13"^^xsd:date .
Class ProductType
Comment: Product types form an irregular subsumption hierarchy (depth 3-5).
Example RDF Instance:
bsbm-inst:ProductType011432
rdf:type bsbm:ProductType ;
rdfs:label "Digital Camera" ;
rdfs:comment "A camera that records pictures electronically rather than on film." ;
rdfs:subClassOf bsbm-inst:ProductType011000
dc:publisher bsbm-inst:StandardizationInstitution01 ;
dc:date "2008-02-13"^^xsd:date .
Class ProductFeature
Comment: The set of possible product features for a specific product depends on the product type. Each product type in the hierarchy has a set of associated product features, which leads to some features being very generic and others being more specific.
Example RDF Instance:
bsbm-inst:ProductFeature103433
rdf:type bsbm:ProductFeature ;
rdfs:label "Wide Screen TFT-Display" ;
rdfs:comment "Wide Screen TFT-Display." ;
dc:publisher bsbm-inst:StandardizationInstitution01 ;
dc:date "2008-02-13"^^xsd:date .
Class Producer
Example RDF Instance:
dataFromProducer001411:Producer001411
rdf:type bsbm:Producer ;
rdfs:label "Canon" ;
rdfs:comment "Canon is a world leader in imaging products and solutions for the digital home and office." ;
foaf:homepage <http://www.canon.com/>
bsbm:country <http://downlode.org/rdf/iso-3166/countries#US> ;
dc:publisher dataFromProducer001411:Producer001411 ;
dc:date "2008-02-13"^^xsd:date .
Class Vendor
Example RDF Instance:
dataFromVendor001400:Vendor001400
rdf:type bsbm:Vendor ;
rdfs:label "Cheap Camera Place" ;
rdfs:comment "We sell the cheapest cameras." ;
foaf:homepage <http://www.cameraplace.com/>
bsbm:country <http://downlode.org/rdf/iso-3166/countries#GB> ;
dc:publisher dataFromVendor001400:Vendor001400 ;
dc:date "2008-02-03"^^xsd:date .
Class Offer
Example RDF Instance:
dataFromVendor001400:Offer2413
rdf:type bsbm:Offer ;
bsbm:product dataFromProducer001411:Product00001435443 ;
bsbm:vendor dataFromVendor001400:Vendor001400 ;
bsbm:price "31.99"^^bsbm:USD ;
bsbm:validFrom "2008-02-12"^^xsd:date ;
bsbm:validTo "2008-02-20"^^xsd:date ;
bsbm:deliveryDays "7"^^xsd:Integer ;
bsbm:offerWebpage <http://vendor001400.com/offers/Offer2413>
dc:publisher dataFromVendor001400:Vendor001400 ;
dc:date "2008-02-13"^^xsd:date .
Class Person
Example RDF Instance:
dataFromRatingSite0014:Reviewer1213
rdf:type foaf:Person ;
foaf:name "Jenny324" ;
foaf:mbox_sha1sum "4749d7c44dc4c0adf66c1319d42b89e18df6df76" ;
bsbm:country <http://downlode.org/rdf/iso-3166/countries#DE> ;
dc:publisher dataFromRatingSite0014:RatingSite0014 ;
dc:date "2007-10-13"^^xsd:date .
Class Review
Example RDF Instance:
dataFromRatingSite0014:Review022343
rdf:type rev:Review ;
bsbm:reviewFor dataFromProducer001411:Product00001435443 ;
rev:reviewer dataFromRatingSite0014:Reviewer1213 ;
bsbm:reviewDate "2007-10-10"^^xsd:date ;
dc:title "This is a nice small camera"@en ;
rev:text "Open your wallet, take out a credit card. No, I'm not going to ask you to order one just yet ..."@en
bsbm:rating1 "5"^^xsd:Integer ;
bsbm:rating2 "4"^^xsd:Integer ;
bsbm:rating3 "3"^^xsd:Integer ;
bsbm:rating4 "4"^^xsd:Integer ;
dc:publisher dataFromRatingSite0014:RatingSite0014 ;
dc:date "2007-10-13"^^xsd:date .
In order to compare the performance of systems that expose SPARQL endpoints, but use different internal data models, there are three different representations of the benchmark dataset as well as different versions of the benchmark queries.
Within the triple representation of the dataset, the publisher and the publication data is captured for each instance by a dc:publisher and a dc:date triple.
Examples:
dataFromVendor001400:Offer2413
rdf:type bsbm:Offer ;
bsbm:product dataFromProducer001411:Product00001435443 ;
bsbm:vendor dataFromVendor001400:Vendor001400 ;
bsbm:price "31.99"^^bsbm:USD ;
bsbm:validFrom "2008-02-12"^^xsd:date ;
bsbm:validTo "2008-02-20"^^xsd:date ;
bsbm:deliveryDays "7"^^xsd:Integer ;
bsbm:offerWebpage <http://vendor001400.com/offers/Offer2413>
dc:publisher dataFromVendor001400:Vendor001400 ;
dc:date "2008-02-13"^^xsd:date .
dataFromVendor001400:Offer2414
rdf:type bsbm:Offer ;
bsbm:product dataFromProducer001411:Product00001435444 ;
bsbm:vendor dataFromVendor001400:Vendor001400 ;
bsbm:price "23.99"^^bsbm:USD ;
bsbm:validFrom "2008-02-10"^^xsd:date ;
bsbm:validTo "2008-02-22"^^xsd:date ;
bsbm:deliveryDays "7"^^xsd:Integer ;
bsbm:offerWebpage <http://vendor001400.com/offers/Offer2414>
dc:publisher dataFromVendor001400:Vendor001400 ;
dc:date "2008-02-13"^^xsd:date .
Within the Named Graph version of the dataset, all information that originates from a specific producer, vendor or rating site is put into a distinct named graph. There is one additional graph that contains provenance information (dc:publisher, dc:date) for all other graphs.
Example (using the TriG syntax):
dataFromVendor001400:Graph-2008-02-13 {
dataFromVendor001400:Offer2413
rdf:type bsbm:Offer ;
bsbm:product dataFromProducer001411:Product00001435443 ;
bsbm:vendor dataFromVendor001400:Vendor001400 ;
bsbm:price "31.99"^^bsbm:USD ;
bsbm:validFrom "2008-02-12"^^xsd:date ;
bsbm:validTo "2008-02-20"^^xsd:date ;
bsbm:deliveryDays "7"^^xsd:Integer ;
bsbm:offerWebpage <http://vendor001400.com/offers/Offer2413>
dataFromVendor001400:Offer2414
rdf:type bsbm:Offer ;
bsbm:product dataFromProducer001411:Product00001435444 ;
bsbm:vendor dataFromVendor001400:Vendor001400 ;
bsbm:price "23.99"^^bsbm:USD ;
bsbm:validFrom "2008-02-10"^^xsd:date ;
bsbm:validTo "2008-02-22"^^xsd:date ;
bsbm:deliveryDays "7"^^xsd:Integer ;
bsbm:offerWebpage <http://vendor001400.com/offers/Offer2414>
}
localhost:provenanceData {
dataFromVendor001400:Graph-2008-02-13 dc:publisher dataFromVendor001400:Vendor001400 ;
dataFromVendor001400:Graph-2008-02-13 dc:date "2008-02-13"^^xsd:date .
}
In order to benchmark systems that map relational databases to RDF and rewrite SPARQL queries into SQL queries against an application specific relational data model, the BSBM data generator is also able to output the benchmark dataset as an MySQL dump.
This dump uses the following relational schema:
ProductFeature(nr, label, comment, publisher, publishDate)
ProductType(nr, label, comment, parent, publisher, publishDate)
Producer(nr, label, comment, homepage, country, publisher, publishDate)
Product(nr, label, comment, producer, propertyNum1, propertyNum2, propertyNum3, propertyNum4, propertyNum5,
propertyNum6, propertyTex1, propertyTex2, propertyTex3, propertyTex4, propertyTex5, propertyTex6,
publisher, publishDate)
ProductTypeProduct(product, productType)
ProductFeatureProduct(product, productFeature)
Vendor(nr, label, comment, homepage, country, publisher, publishDate)
Offer(nr, product, producer, vendor, price, validFrom, validTo, deliveryDays, offerWebpage, publisher, publishDate)
Person(nr, name, mbox_sha1sum, country, publisher, publishDate)
Review(nr, product, producer, person, reviewDate, title, text, language, rating1, rating2, rating3, rating4,
publisher, publishDate)
This section defines the rules for generating benchmark data for a given scale factor.
The benchmark is scaled by the number of products.
The table below gives an overview about the characteristics of BSBM datasets with different scale factors.
| Scale Factor | 666 | 2,785 | 70,812 | 284,826 |
|
Number
of RDF Triples
|
250K | 1M | 25M | 100M |
| Number of Producers | 14 | 60 | 1422 | 5,618 |
| Number of Product Features | 2,860 | 4,745 | 23,833 | 47,884 |
| Number of Product Types | 55 | 151 | 731 | 2011 |
| Number of Vendors | 8 | 34 | 722 | 2,854 |
| Number of Offers | 13,320 | 55,700 | 1,416,240 | 5,696,520 |
| Number of Reviewers | 339 | 1432 | 36,249 | 146,054 |
| Number of Reviews | 6,660 | 27,850 | 708,120 | 2,848,260 |
| Total Number of Instances |
23,922
|
92,757
|
2,258,129
|
9,034,027
|
| Exact Total Number of Triples | 250,030 | 1,000,313 | 25,000,244 | 100,000,112 |
| File Size Turtle (unzipped) | 22 MB | 86 MB | 2.1 GB | 8.5 GB |
The BSBM data generator is described in Section 8.
Products have product types and are described with various properties. There are products with several different product property combinations (many properties, less properties).
Rules for data generation:
There are three types of product descriptions. The table below lists the textual and numeric properties for each type.
| Textual Properties | Numeric Properties | ||
| Description Type 1 | PropertyTextual1 to PropertyTextual5 | PropertyNumeric1 to PropertyNumeric5 | 40% |
| Description Type 2 |
PropertyTextual1 to PropertyTextual3 |
PropertyNumeric1 to PropertyNumeric3 + optional PropertyNumeric4 (50%) + optional PropertyNumeric5 (25%) |
20% |
| Description Type 3 |
PropertyTextual1 to PropertyTextual3 |
PropertyNumeric1 to PropertyNumeric3 + optional PropertyNumeric5 (25%) + optional PropertyNumeric6 (50%) |
40% |
Relation: Product-Producer
Relation: Product-ProductType
Relation: Product-ProductFeature
Relation: Product-Offer
Relation: Product-Review
Irregular subsumption hierarchy (depth 2-6). Number of classes increases with the number of products (around 4log10#Products).
The branching factor for every node on the same level is equal and gets calculated for arbitrary scale factors. The table below illustrates the relationship between number of products and branching factors for every level:
| root level | level 1 | level 2 | level 3 | level 4 | |
| 100 products | 4 | 4 | |||
| 1 000 products | 6 | 8 | 2 | ||
| 10 000 products | 8 | 8 | 4 | ||
| 100 000 products | 10 | 8 | 8 | 2 | |
| 1 000 000 products | 12 | 8 | 8 | 4 | |
| 10 000 000 products | 14 | 8 | 8 | 8 | 2 |
| 100 000 000 products | 16 | 8 | 8 | 8 | 4 |
As can be seen the depth increases by one everytime the product count grows by a factor of 100.
Rules for data generation:Each feature is assigned to a product type in the type hierarchy, which leads to some features being very generic and other being more specific.
Rules for data generation:
Rules for data generation:
Per 1000 products, there are 20 producers generated on average.
Per 1000 products, there is 0.5 vendor generated on average.
Relation: Vendor-OfferRules for data generation:
Per 1000 products, there are 20000 offers generated.
Rules for data generation:
Per 1000 products, there are on average 500 persons generated.
Rules for data generation:
Per 1000 products, there are 10000 reviews generated.
Relation: Review-Person
Relation: Review-Ratingsite
Dictionary 1: Words from set of product names (around 90.000 words)
Dictionary 2: Words from English text (todo: look for a corpus with English sentences, currently dictionary 1 is used)
Dictionary 3: Names of persons (around 90.000 names)
This section defines a suite of benchmark queries and a query mix.
The benchmark queries are designed to emulate the search and navigation pattern of a consumer looking for a product. A product search includes the following steps:
There are three representations of the benchmark query set: One for the Triple and one for the Named Graphs data model as well as a pure SQL version for the relational representation given in section 2.2.4. All query sets have the same semantics.
There are two variations of the BSBM query mix:
The complete query mix consists of 25 queries that simulate a product search by a single consumer. The query sequenze is given below:
The reduced query mix consists of the same sequence as the complete query mix but without:
10. Query 5: Find products that are similar to a given product.
13. Query 6: Find products having a label that contains a specific
string.
Query 5 and 6 are excluded from the reduced query mix, as they consume together about 80% of the overall execution time on current stores when executed against larger datasets. As of 2008, we therefore recomment to report performance figures for the reduced query mix in addition to figures for the complete query mix, if the benchmark is run against datasets above 25 million triples.
Each query is defined by the following components:
Use Case Motivation: A consumer is looking for a product and has a general idea about what he wants.
SPARQL Query:
PREFIX bsbm-inst: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/>
PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT DISTINCT ?product ?label
WHERE {
?product rdfs:label ?label .
?product a %ProductType% .
?product bsbm:productFeature %ProductFeature1% .
?product bsbm:productFeature %ProductFeature2% .
?product bsbm:productPropertyNumeric1 ?value1 .
FILTER (?value1 > %x%)
}
ORDER BY ?label
LIMIT 10
Parameters:
| Parameter | Description |
|---|---|
| %ProductType% | A randomly selected Class URI from the class hierarchy (one level above leaf level). |
| %ProductFeature1% %ProductFeature2% |
Two different, randomly selected feature URIs that correspond to the chosen product type. |
| %x% | A number between 1 and 500 |
Query Properties:
Use Case Motivation: The consumer wants to view basic information about products found by query 1.
SPARQL Query
PREFIX bsbm-inst: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/>
PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
SELECT ?label ?comment ?producer ?productFeature ?propertyTextual1 ?propertyTextual2 ?propertyTextual3
?propertyNumeric1 ?propertyNumeric2 ?propertyTextual4 ?propertyTextual5 ?propertyNumeric4
WHERE {
%ProductXYZ% rdfs:label ?label .
%ProductXYZ% rdfs:comment ?comment .
%ProductXYZ% bsbm:producer ?p .
?p rdfs:label ?producer .
%ProductXYZ% dc:publisher ?p .
%ProductXYZ% bsbm:productFeature ?f .
?f rdfs:label ?productFeature .
%ProductXYZ% bsbm:productPropertyTextual1 ?propertyTextual1 .
%ProductXYZ% bsbm:productPropertyTextual2 ?propertyTextual2 .
%ProductXYZ% bsbm:productPropertyTextual3 ?propertyTextual3 .
%ProductXYZ% bsbm:productPropertyNumeric1 ?propertyNumeric1 .
%ProductXYZ% bsbm:productPropertyNumeric2 ?propertyNumeric2 .
OPTIONAL { %ProductXYZ% bsbm:productPropertyTextual4 ?propertyTextual4 }
OPTIONAL { %ProductXYZ% bsbm:productPropertyTextual5 ?propertyTextual5 }
OPTIONAL { %ProductXYZ% bsbm:productPropertyNumeric4 ?propertyNumeric4 }
}
Parameters:
| Parameter | Description |
|---|---|
| %ProductXYZ% | A product URI (randomly selected) |
Query Properties:
Use Case Motivation: After looking at information about some products, the consumer has a more specific idea what we wants. Therefore, he asks for products having several features but not having a specific other feature.
SPARQL Query:
PREFIX bsbm-inst: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/>
PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT ?product ?label
WHERE {
?product rdfs:label ?label .
?product a %ProductType% .
?product bsbm:productFeature %ProductFeature1% .
?product bsbm:productPropertyNumeric1 ?p1 .
FILTER ( ?p1 > %x% )
?product bsbm:productPropertyNumeric3 ?p3 .
FILTER (?p3 < %y% )
OPTIONAL {
?product bsbm:productFeature %ProductFeature2% .
?product rdfs:label ?testVar }
FILTER (!bound(?testVar))
}
ORDER BY ?label
LIMIT 10
Parameters:
| Parameter | Description |
|---|---|
| %ProductType% | A randomly selected Class URI from the class hierarchy (leaf level). |
| %ProductFeature1% %ProductFeature2% |
Three different, randomly selected product feature URI that correspond to the chosen product type. |
| %x% %y% |
Two random numbers between 1 and 500 |
Query Properties:
Use Case Motivation: After looking at information about some products, the consumer has a more specific idea what we wants. Therefore, he asks for products matching either one set of features or another set.
SPARQL Query:
PREFIX bsbm-inst: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/>
PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
SELECT DISTINCT ?product ?label ?propertyTextual
WHERE {
{
?product rdfs:label ?label .
?product rdf:type %ProductType% .
?product bsbm:productFeature %ProductFeature1% .
?product bsbm:productFeature %ProductFeature2% .
?product bsbm:productPropertyTextual1 ?propertyTextual .
?product bsbm:productPropertyNumeric1 ?p1 .
FILTER ( ?p1 > %x% )
} UNION {
?product rdfs:label ?label .
?product rdf:type %ProductType% .
?product bsbm:productFeature %ProductFeature1% .
?product bsbm:productFeature %ProductFeature3% .
?product bsbm:productPropertyTextual1 ?propertyTextual .
?product bsbm:productPropertyNumeric2 ?p2 .
FILTER ( ?p2> %y% )
}
}
ORDER BY ?label
OFFSET 5
LIMIT 10
Parameters:
| Parameter | Description |
|---|---|
| %ProductType% | A randomly selected Class URI from the class hierarchy (leaf level). |
| %ProductFeature1% %ProductFeature2% %ProductFeature3% |
Three different, randomly selected product feature URI that correspond to the chosen product type. |
| %x% %y% |
Two random numbers between 1 and 500 |
Query Properties:
Use Case Motivation: The consumer has found a product that fulfills his requirements. He now wants to find products with similar features.
SPARQL Query:
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
SELECT DISTINCT ?product ?productLabel
WHERE {
?product rdfs:label ?productLabel .
FILTER (%ProductXYZ% != ?product)
%ProductXYZ% bsbm:productFeature ?prodFeature .
?product bsbm:productFeature ?prodFeature .
%ProductXYZ% bsbm:productPropertyNumeric1 ?origProperty1 .
?product bsbm:productPropertyNumeric1 ?simProperty1 .
FILTER (?simProperty1 < (?origProperty1 + 120) && ?simProperty1 > (?origProperty1 – 120))
%ProductXYZ% bsbm:productPropertyNumeric2 ?origProperty2 .
?product bsbm:productPropertyNumeric2 ?simProperty2 .
FILTER (?simProperty2 < (?origProperty2 + 170) && ?simProperty2 > (?origProperty2 – 170))
}
ORDER BY ?productLabel
LIMIT 5
Parameters:
| Parameter | Description |
|---|---|
| %ProductXYZ% | A product URI (randomly selected) |
Query Properties:
Use Case Motivation: The consumer remembers parts of a product name from former searches. He wants to find the product again by searching for the parts of the name that he remembers.
SPARQL Query:
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
SELECT ?product ?label
WHERE {
?product rdfs:label ?label .
?product rdf:type bsbm:Product .
FILTER regex(?label, "%word1%")
}
Parameters:
| Parameter | Description |
|---|---|
| %word1% |
A word from the list of words that were used in the dataset generation. |
Query Properties:
Use Case Motivation: The consumer has found a products which fulfills his requirements. Now he wants in-depth information about this product including offers from German vendors and product reviews if existent.
SPARQL Query:
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX rev: <http://purl.org/stuff/rev#>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
SELECT ?productLabel ?offer ?price ?vendor ?vendorTitle ?review ?revTitle
?reviewer ?revName ?rating1 ?rating2
WHERE {
%ProductXYZ% rdfs:label ?productLabel .
OPTIONAL {
?offer bsbm:product %ProductXYZ% .
?offer bsbm:price ?price .
?offer bsbm:vendor ?vendor .
?vendor rdfs:label ?vendorTitle .
?vendor bsbm:country <http://downlode.org/rdf/iso-3166/countries#DE> .
?offer dc:publisher ?vendor .
?offer bsbm:validTo ?date .
FILTER (?date > %currentDate% )
}
OPTIONAL {
?review bsbm:reviewFor %ProductXYZ% .
?review rev:reviewer ?reviewer .
?reviewer foaf:name ?revName .
?review dc:title ?revTitle .
OPTIONAL { ?review bsbm:rating1 ?rating1 . }
OPTIONAL { ?review bsbm:rating2 ?rating2 . }
}
}
Parameters:
| Parameter | Description |
|---|---|
| %ProductXYZ% | A product URI (randomly selected) |
| %currentDate% | A date within the validFrom validTo range of the offers (same date for all queries within a run). |
Query Properties:
Use Case Motivation: The consumer wants to read the 20 most recent English language reviews about a specific product.
SPARQL Query:
PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
PREFIX rev: <http://purl.org/stuff/rev#>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
SELECT ?title ?text ?reviewDate ?reviewer ?reviewerName ?rating1 ?rating2 ?rating3 ?rating4
WHERE {
?review bsbm:reviewFor %ProductXYZ% .
?review dc:title ?title .
?review rev:text ?text .
FILTER langMatches( lang(?text), "EN" )
?review bsbm:reviewDate ?reviewDate .
?review rev:reviewer ?reviewer .
?reviewer foaf:name ?reviewerName .
OPTIONAL { ?review bsbm:rating1 ?rating1 . }
OPTIONAL { ?review bsbm:rating2 ?rating2 . }
OPTIONAL { ?review bsbm:rating3 ?rating3 . }
OPTIONAL { ?review bsbm:rating4 ?rating4 . }
}
ORDER BY DESC(?reviewDate)
LIMIT 20
Parameters:
| Parameter | Description |
|---|---|
| %ProductXYZ% | A product URI (randomly selected) |
Query Properties:
Use Case Motivation: In order to decide whether to trust a review, the consumer asks for any kind of information that is available about the reviewer.
SPARQL Query:
PREFIX rev: <http://purl.org/stuff/rev#>
DESCRIBE ?x
WHERE { %ReviewXYZ% rev:reviewer ?x }
Parameters:
| Parameter | Description |
|---|---|
| %ReviewXYZ% | A review URI (randomly selected) |
Query Properties:
Use Case Motivation: The consumer wants to buy from a vendor in the United States that is able to deliver within 3 days and is looking for the cheapest offer that fulfills these requirements.
SPARQL Query:
PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
SELECT DISTINCT ?offer ?price
WHERE {
?offer bsbm:product %ProductXYZ% .
?offer bsbm:vendor ?vendor .
?offer dc:publisher ?vendor .
?vendor bsbm:country <http://downlode.org/rdf/iso-3166/countries#US> .
?offer bsbm:deliveryDays ?deliveryDays .
FILTER (?deliveryDays <= 3)
?offer bsbm:price ?price .
?offer bsbm:validTo ?date .
FILTER (?date > %currentDate% )
}
ORDER BY xsd:double(str(?price))
LIMIT 10
Parameters:
| Parameter | Description |
|---|---|
| %ProductXYZ% | A product URI (randomly selected) |
| %currentDate% | A date within the validFrom-validTo range of the offers (same date for all queries within a run). |
Query Properties:
Use Case Motivation: After deciding on a specific offer, the consumer wants to get all information that is directly related to this offer.
SPARQL Query:
SELECT ?property ?hasValue ?isValueOf
WHERE {
{ %OfferXYZ% ?property ?hasValue }
UNION
{ ?isValueOf ?property %OfferXYZ% }
}
Parameters:
| Parameter | Description |
|---|---|
| %OfferXYZ% | An offer URI (randomly selected) |
Query Properties:
Use Case Motivation: After deciding on a specific offer, the consumer wants to save information about this offer on his local machine using a different RDF schema.
SPARQL Query:
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX rev: <http://purl.org/stuff/rev#>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
PREFIX bsbm-export: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/export/>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
CONSTRUCT { %OfferXYZ% bsbm-export:product ?productURI .
%OfferXYZ% bsbm-export:productlabel ?productlabel .
%OfferXYZ% bsbm-export:vendor ?vendorname .
%OfferXYZ% bsbm-export:vendorhomepage ?vendorhomepage .
%OfferXYZ% bsbm-export:offerURL ?offerURL .
%OfferXYZ% bsbm-export:price ?price .
%OfferXYZ% bsbm-export:deliveryDays ?deliveryDays .
%OfferXYZ% bsbm-export:validuntil ?validTo }
WHERE { %OfferXYZ% bsbm:product ?productURI .
?productURI rdfs:label ?productlabel .
%OfferXYZ% bsbm:vendor ?vendorURI .
?vendorURI rdfs:label ?vendorname .
?vendorURI foaf:homepage ?vendorhomepage .
%OfferXYZ% bsbm:offerWebpage ?offerURL .
%OfferXYZ% bsbm:price ?price .
%OfferXYZ% bsbm:deliveryDays ?deliveryDays .
%OfferXYZ% bsbm:validTo ?validTo }
Parameters:
| Parameter | Description |
|---|---|
| %OfferXYZ% | An offer URI (randomly selected) |
Query Properties:
The queries for the Named Graphs data model have the same semantics as the queries for the triple data model. The queries do not specify the IRIs of the named graphs in the RDF Dataset using the FROM NAMED clause, but assume that the query is executed against the complete RDF Dataset.
This is still work in progress ...
Todo: Rewrite all queries for Named Graphs. Two examples are already
found below:
SPARQL Query
PREFIX bsbm-inst: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/instances/>
PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT ?label ?comment ?producer ?productFeature ?propertyTextual1 ?propertyTextual2
?propertyNumeric1 ?propertyNumeric2 ?propertyTextual4 ?propertyTextual5 ?propertyNumeric4
WHERE {
GRAPH ?graph {
%ProductXYZ% rdfs:label ?label .
%ProductXYZ% rdfs:comment ?comment .
%ProductXYZ% bsbm:producer ?p .
?p rdfs:label ?producer .
%ProductXYZ% bsbm:productFeature ?f .
?f rdfs:label ?productFeature .
%ProductXYZ% bsbm:productPropertyTextual1 ?propertyTextual1 .
%ProductXYZ% bsbm:productPropertyTextual2 ?propertyTextual2 .
%ProductXYZ% bsbm:productPropertyNumeric1 ?propertyNumeric1 .
%ProductXYZ% bsbm:productPropertyNumeric2 ?propertyNumeric2 .
OPTIONAL { %ProductXYZ% bsbm:productPropertyTextual4 ?propertyTextual4 }
OPTIONAL { %ProductXYZ% bsbm:productPropertyTextual5 ?propertyTextual5 }
OPTIONAL { %ProductXYZ% bsbm:productPropertyNumeric4 ?propertyNumeric4 }
}
GRAPH localhost:provenanceData {
?graph dc:publisher ?p .
}
}
SPARQL Query:
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX rev: <http://purl.org/stuff/rev#>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX bsbm: <http://www4.wiwiss.fu-berlin.de/bizer/bsbm/v01/vocabulary/>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
SELECT ?productLabel ?offer ?price ?vendor ?vendorTitle ?review ?revTitle
?reviewer ?revName ?rating1 ?rating2
WHERE {
GRAPH ?producerGraph {
%ProductXYZ% rdfs:label ?productLabel .
}
OPTIONAL {
GRAPH ?vendorGraph {
?offer bsbm:product %ProductXYZ% .
?offer bsbm:price ?price .
?offer bsbm:vendor ?vendor .
?vendor rdfs:label ?vendorTitle .
?offer bsbm:validTo ?date .
FILTER (?date > %currentDate% )
}
}
OPTIONAL {
GRAPH ?ratingSiteGraph {
?review bsbm:reviewFor %ProductXYZ% .
?review rev:reviewer ?reviewer .
?reviewer foaf:name ?revName .
?review dc:title ?revTitle .
OPTIONAL { ?review bsbm:rating1 ?rating1 . }
OPTIONAL { ?review bsbm:rating2 ?rating2 . }
}
}
GRAPH localhost:provenanceData {
?vendorGraph dc:publisher ?vendor .
}
}
This section will contain a SQL representation of the benchmark queries in order to be able to compare the performance of stores that expose SPARQL endpoints to the performance of classic SQL-based RDBMS. Since there is no exact counterpart to some SPARQL specific query forms like DESCRIBE, the SQL queries aren't semantically completely equivalent.
Use Case Motivation: A consumer is looking for a product and has a general idea about what he wants.
SQL Query:
SELECT distinct nr, label
FROM product p, producttypeproduct ptp
WHERE p.nr = ptp.product AND ptp.productType=@ProductType@
AND propertyNum1 > @x@
AND p.nr IN (SELECT distinct product FROM productfeatureproduct WHERE productFeature=@ProductFeature1@)
AND p.nr IN (SELECT distinct product FROM productfeatureproduct WHERE productFeature=@ProductFeature2@)
ORDER BY label
LIMIT 10;
Parameters:
| Parameter | Description |
|---|---|
| @ProductType@ | A randomly selected Class ID from the class hierarchy (one level above leaf level). |
| @ProductFeature1@ @ProductFeature2@ |
Two different, randomly selected feature IDs that correspond to the chosen product type. |
| @x@ | A number between 1 and 500 |
Use Case Motivation: The consumer wants to view basic information about products found by query 1.
SQL Query
SELECT pt.label, pt.comment, pt.producer, productFeature, propertyTex1, propertyTex2, propertyTex3,
propertyNum1, propertyNum2, propertyTex4, propertyTex5, propertyNum4
FROM product pt, producer pr, productfeatureproduct pfp
WHERE pt.nr=@ProductXYZ@ AND pt.nr=pfp.product AND pt.producer=pr.nr;
Parameters:
| Parameter | Description |
|---|---|
| @ProductXYZ@ | A product ID (randomly selected) |
Use Case Motivation: After looking at information about some products, the consumer has a more specific idea what we wants. Therefore, he asks for products having several features but not having a specific other feature.
SQL Query:
SELECT p.nr, p.label
FROM product p, producttypeproduct ptp
WHERE p.nr=ptp.product
AND productType=@ProductType@
AND propertyNum1>@x@
AND propertyNum3<@y@
AND @ProductFeature1@ IN (SELECT productFeature FROM productfeatureproduct WHERE product=p.nr)
AND @ProductFeature2@ NOT IN (SELECT productFeature FROM productfeatureproduct WHERE product=p.nr)
ORDER BY p.label
LIMIT 10;
Parameters:
| Parameter | Description |
|---|---|
| @ProductType@ | A randomly selected Class ID from the class hierarchy (leaf level). |
| @ProductFeature1@ @ProductFeature2@ |
Three different, randomly selected product feature ID that correspond to the chosen product type. |
| @x@ @y@ |
Two random numbers between 1 and 500 |
Use Case Motivation: After looking at information about some products, the consumer has a more specific idea what we wants. Therefore, he asks for products matching either one set of features or another set.
SQL Query:
SELECT distinct p.nr, p.label, p.propertyTex1
FROM product p, producttypeproduct ptp
WHERE p.nr=ptp.product AND ptp.productType=@ProductType@
AND p.nr IN (SELECT distinct product FROM productfeatureproduct WHERE productFeature=@ProductFeature1@)
AND ((propertyNum1>@x@ AND p.nr IN (SELECT distinct product FROM productfeatureproduct WHERE productFeature=@ProductFeature2@)
) OR (propertyNum2>@y@ AND p.nr IN (SELECT distinct product FROM productfeatureproduct WHERE productFeature=@ProductFeature3@)))
ORDER BY label
LIMIT 10
OFFSET 5;
Parameters:
| Parameter | Description |
|---|---|
| @ProductType@ | A randomly selected Class ID from the class hierarchy (leaf level). |
| @ProductFeature1@ @ProductFeature2@ @ProductFeature3@ |
Three different, randomly selected product feature IDs that correspond to the chosen product type. |
| @x@ @y@ |
Two random numbers between 1 and 500 |
Use Case Motivation: The consumer has found a product that fulfills his requirements. He now wants to find products with similar features.
SQL Query:
SELECT distinct p.nr, p.label
FROM product p, product po,
(Select distinct pfp1.product FROM productfeatureproduct pfp1, (SELECT productFeature FROM productfeatureproduct WHERE product=@ProductXYZ@) pfp2 WHERE pfp2.productFeature=pfp1.productFeature) pfp
WHERE p.nr=pfp.product AND po.nr=@ProductXYZ@ AND p.nr!=po.nr
AND p.propertyNum1<(po.propertyNum1+120) AND p.propertyNum1>(po.propertyNum1-120)
AND p.propertyNum2<(po.propertyNum2+170) AND p.propertyNum2>(po.propertyNum2-170)
ORDER BY label
LIMIT 5;
Parameters:
| Parameter | Description |
|---|---|
| @ProductXYZ@ | A product ID (randomly selected) |
Use Case Motivation: The consumer remembers parts of a product name from former searches. He wants to find the product again by searching for the parts of the name that he remembers.
SQL Query:
SELECT nr, label
FROM product
WHERE label like "%@word1@%";
Parameters:
| Parameter | Description |
|---|---|
| @word1@ |
A word from the list of words that were used in the dataset generation. |
Use Case Motivation: The consumer has found a products which fulfills his requirements. Now he wants in-depth information about this product including offers from German vendors and product reviews if existent.
SQL Query:
SELECT *
FROM (select label from product where nr=@ProductXYZ@) p left join
((select o.nr as onr, o.price, v.nr as vnr, v.label from offer o, vendor v where @ProductXYZ@=o.product AND
o.vendor=v.nr AND v.country='DE' AND o.validTo>'@currentDate@') ov right join
(select r.nr as rnr, r.title, pn.nr as pnnr, pn.name, r.rating1, r.rating2 from review r, person pn where r.product=@ProductXYZ@ AND
r.person=pn.nr) rpn on (1=1)) on (1=1);
Parameters:
| Parameter | Description |
|---|---|
| @ProductXYZ@ | A product ID (randomly selected) |
| @currentDate@ | A date within the validFrom validTo range of the offers (same date for all queries within a run). |
Use Case Motivation: The consumer wants to read the 20 most recent English language reviews about a specific product.
SQL Query:
SELECT r.title, r.text, r.reviewDate, p.nr, p.name, r.rating1, r.rating2, r.rating3, r.rating4
FROM review r, person p
WHERE r.product=@ProductXYZ@ AND r.person=p.nr
AND r.language='en'
ORDER BY r.reviewDate desc
LIMIT 20;
Parameters:
| Parameter | Description |
|---|---|
| @ProductXYZ@ | A product ID (randomly selected) |
Use Case Motivation: In order to decide whether to trust a review, the consumer asks for any kind of information that is available about the reviewer.
SQL Query:
SELECT p.nr, p.name, p.mbox_sha1sum, p.country, r2.nr, r2.product, r2.title
FROM review r, person p, review r2
WHERE r.nr=@ReviewXYZ@ AND r.person=p.nr AND r2.person=p.nr;
Parameters:
| Parameter | Description |
|---|---|
| @ReviewXYZ@ | A review ID (randomly selected) |
Use Case Motivation: The consumer wants to buy from a vendor in the United States that is able to deliver within 3 days and is looking for the cheapest offer that fulfills these requirements.
SQL Query:
SELECT distinct o.nr, o.price
FROM offer o, vendor v
WHERE o.product=@ProductXYZ@
AND o.deliveryDays<=3 AND v.country='US'
AND o.validTo>'@currentDate@' AND o.vendor=v.nr
Order BY o.price
LIMIT 10;
Parameters:
| Parameter | Description |
|---|---|
| @ProductXYZ@ | A product ID (randomly selected) |
| @currentDate@ | A date within the validFrom-validTo range of the offers (same date for all queries within a run). |
Use Case Motivation: After deciding on a specific offer, the consumer wants to get all information that is directly related to this offer.
SQL Query:
Select product, producer, vendor, price, validFrom, validTo, deliveryDays, offerWebpage, publisher, publishDate
from offer
where nr=@OfferXYZ@;
Parameters:
| Parameter | Description |
|---|---|
| @OfferXYZ@ | An offer ID (randomly selected) |
Use Case Motivation: After deciding on a specific offer, the consumer wants to save information about this offer on his local machine using a different RDF schema.
SQL Query:
Select p.nr As productNr, p.label As productlabel, v.label As vendorname, v.homepage As vendorhomepage,
o.offerWebpage As offerURL, o.price As price, o.deliveryDays As deliveryDays, o.validTo As validTo
From offer o, product p, vendor v
Where o.nr=@OfferXYZ@ AND o.product=p.nr AND o.vendor=v.nr;
Parameters:
| Parameter | Description |
|---|---|
| @OfferXYZ@ | An offer ID (randomly selected) |
The three fundamental performance metrics of the BSBM are:
4.1.1 Metrics for Single Queries
Average Query Execution Time (aQET): Average time for executing an individual query of type x multiple times with different parameters against the SUT.
Queries per Second (QpS): Average amount of queries of type x that were executed per second.
Min/Max Query Execution Time (minQET, maxQET): A lower and upper bound execution time for queries of type x.
4.1.2 Metrics for Query Mixes
Queries Mixes per Hour (QMpH): Number of query mixes with different parameters that are executed per hour against the SUT.
Overall Runtime (oaRT): Overall time it took the test driver to execute a certain amount of query mixes against the SUT.
Composite Query Execution Time (cQET): Average time for executing the query mix multiple times with different parameters.
Average Query Execution Time over all Queries (aQEToA): Overall time to run 50 query mixes devided by the number of queries (25*50=1250).
4.1.3 Price/Performance Metric for the Complete System under Test (SUT)
The Price/Performance Metric defined as $ / QMpH.
Where $ is the total system cost over 5 years in the specified currency. The total system cost over 5 years is calculated according to the TPC Pricing Specification. If compute on demand infrastructure is used, the costing will be $/QMpH/day.
When running the BSBM benchmark and reporting BSBM benchmark results, you should obey to the following rules:
The Berlin SPARQL Benchmark is designed to support the comparison of native RDF stores, native Named Graph stores, systems that map relational databases into RDF, and SPARQL-wrappers around other kinds of data sources.
There are the following benchmark scenarios:
Scenario for benchmarking native triple stores.
Acronym : NTR = Native Triple Repository
Setup: One or more clients ask SPARQL queries over the SPARQL protocol against a single store which exposes a triple data model.
Use Case Motivation: An e-shop selling different products provides a SPARQL endpoint over its data.
Scenario for benchmarking native Named Graphs stores.
Acronym: NNGR = Native Named Graphs Repository
Setup: One or more clients ask SPARQL queries over the SPARQL protocol against a single store which exposes a Named Graphs data model.
Use Case Motivation: An electronic market which integrates information from different producers, vendors and rating sites provides a SPARQL endpoint over its data.
Scenario for benchmarking other types of data sources that expose triple-based SPARQL endpoints. For instance, RDF-mapped relational databases, wrappers around WebAPIs and the like.
Acronym: MTS = Mapped Triple Source
Setup: One or more clients ask SPARQL queries over the SPARQL protocol against a store which exposes a triple data model.
Use Case Motivation: An e-shop selling different products provides a SPARQL endpoint over its data.
Benchmark Dataset:
The data generator can export data in a generic XML format and as MySQL dump, which can be imported into different stores, for instance relational data bases.
Scenario for benchmarking other types of data sources that expose named graph-based SPARQL endpoints. For instance, RDF-mapped relational databases, wrappers around WebAPIs or the like.
Acronym: MNGS = Mapped Named Graph Source
Setup: One or more clients ask SPARQL queries over the SPARQL protocol against a store which exposes a named graph-based data model.
Use Case Motivation: An electronic market which integrates information from different producers, vendors and rating sites provides a SPARQL endpoint over its data.
Benchmark Dataset:
The data generator can export data in a generic XML format and as MySQL dump, which can be imported into different stores, for instance relational data bases.
This section defines formats for reporting benchmark results.
Benchmark results are named according to the scenario, the
scale factor of the dataset and the number of concurrent clients.
For example:
NTP(1000,5) means
23.7 QPS(2)-NTS(10000,1)
means that on average 23.7 queries of type 2 were executed per second by a single client stream against a Native Triple Store containing data about 10,000 products.
To guarantee an intelligible interpretation of benchmark reports/results as well as to allow for efficient and even automated handling/comparisons, all necessary information shall be represented in XML. Furthermore we opt for full disclosure policy of the SUT, configuration, pricing etc. to give all information needed for replicating any detail of the system, thus enabling anyone to achieve similar benchmark results.
Full Disclosure Report Contents
Todo: Define an XML format for the Full
Disclosure Report
Todo: Implement a nice tool which generates HTML reports including nice
graphics from XML benchmark
results in order to motivate people to use the reporting format.
Before the performance of a SUT is measured, it has to be verified that the SUT returns correct results for the benchmark queries.
For testing whether a SUT returns correct results, the BSBM benchmark provides a qualification dataset and a qualification tool which compares the query results of a SUT with the correct query results. At the moment, the
qualification
tool verifies only
the results of SELECT queries. The results of DESCRIBE and CONSTRICT queries (queries 9 and 12) are not checked.
A BSBM qualification test is conducted in the two-step procedure described below:
$ java -cp bin:lib/* benchmark.testdriver.TestDriver -q http://SUT/sparqlThis will create a qualification file named "run.qual" (different file name can be specified with the "-qf" parameter) which is used in step 2. Also the run.log (if logging is set to "ALL" in the log4j.xml file) contains all queries with full result text, so single queries can be examined later on.
where http://SUT/sparql specifies the SPARQL endpoint
| Option | Description |
| -rc | Only check the amount of results returned and not the result content. |
| -ql <qualification log file name> | Specify the file name to write the qualification test results into. |
$ java -cp bin:lib/* benchmark.qualification.Qualification correct.qual run.qualThis generates by default a log file called "qual.log" with the following content:
where run.qual is the qualification file generated by the Test Driver in qualification mode
There is a Java (at least JVM 1.5 needed) implementation of a data generator and a test driver for the BSBM benchmark.
The source code of the data generator and the test driver can
be downloaded from Sourceforge BSBM tools.
The code is licensed under the terms of the GNU General Public
License.
The BSBM data generator can be used to create benchmark datasets of different sizes. Data generation is deterministic.
The data generator supports the following output formats:
| Format | Option |
| N-Triples | -s nt |
| Turtle | -s ttl |
| XML | -s xml |
| (My-)SQL dump | -s sql |
Next on the todo list: Implement TriG output format for benchmarking Named Graph stores.
Configuration options:
| Option | Description |
|---|---|
| -s <output format> | For the dataset there are several output formats supported. See upper table for details. Default: nt |
| -pc <number of products> |
Scale factor: The dataset is
scaled via the number of products. For example: 91 products make about
50K triples. Default: 100
|
| -fc |
The
data generator by default adds rdf:type statements for all types of a
product to the dataset. If the SUT supports RDFS reasoning, the option
-fc can be used to exclude these statements and leave generating them
to the inference engine of the store. Default: disabled
|
| -dir |
The output directory for all the
data the Test Driver uses for its runs. Default: "td_data"
|
| -fn |
The file name for the generated
dataset (suffix is added according to the output format). Default:
"dataset"
|
The following example command creates a Turtle benchmark dataset with the scale factor 1000 and forward chaining enabled:
$ java -cp bin:lib/ssj.jar benchmark.generator.Generator -fc -pc 1000 -s ttl
The test driver works against a SPARQL endpoint over the SPARQL protocol.
Configuration options:
| Option | Description |
|---|---|
| -runs <number of runs> |
The number of query mix runs.
Default: 50
|
| -idir <directory> |
The input parameter directory
which was created by the Data Generator. Default: "td_data"
|
| -w <number of warm up runs> |
Number of runs executed before the
actual test to warm up the store. Default: 10
|
| -o <result XML file> |
The output file containing the
aggregated result overview. Default: "benchmark_result.xml"
|
| -dg <default graph URI> |
Specify a default
graph for the queries. Default: null
|
| -mt <number of clients> |
Benchmark with multiple concurrent
clients.
|
| -seed <Long value> |
Set the seed for the random number
generator used for the parameter generation.
|
| -t <Timeout in ms> |
If for a specific query the
complete result is not read after the specified timeout, the client
disconnects and reports a timeout to the Test Driver. This is also the
maximum runtime a query can contribute to the metrics.
|
| -q | Turn on qualification mode. For more information, see the qualification chapter. |
| -qf <qualification file name> | Change the qualification file name, also see the qualification chapter. |
In addition to these options a SPARQL-endpoint must be given.
A detailed run log is generated for log level 'ALL' containing information about every executed query.
The following example command runs 128 query mixes (plus 32 for warm-up) against a SUT which provides a SPARQL-endpoint at http://localhost/sparql:
$ java -cp bin:lib/* benchmark.testdriver.TestDriver http://localhost/sparqlThe following example runs 1024 query mixes plus 128 warm up mixes with 4 clients against a SUT which provides a SPARQL-endpoint. The timeout per query is set to 30s.
Or, if your java version does not support the asterisk in the classpath definition, you can write:
java -cp bin:lib/ssj.jar:lib/log4j-1.2.15.jar benchmark.testdriver.TestDriver http://localhost/sparql
$ java -cp bin:lib/ssj.jar:lib/log4j-1.2.15.jar benchmark.testdriver.TestDriver -runs 1024 -w 128 -mt 4 -t 30000 http://localhost/sparql
For more information about RDF and SPARQL Benchmarks please refer to:
Lots of thanks to