Jelly user guide

To use Jelly, pick an implementation that matches your tech stack:

jelly-cli – command-line tool, works on Windows, macOS, and Linux.
Jelly-JVM – written in Java, integrated with Apache Jena, RDF4J, Titanium, and Neo4j.
pyjelly – written in Python, integrated with RDFLib. Can also be used without RDFLib.
jelly_rs (experimental) – written in Rust, integrated with Sophia.
rdfjs-jelly (unofficial, community-led) – written in JavaScript.
Locorda (unofficial, community-led) – written in Dart.

You can also create your own implementation. Because Jelly is built on top of Protocol Buffers, you can generate a lot of the code automatically in any popular programming language.

What can it do?

Jelly is designed to be a protocol for streaming RDF knowledge graphs, but it can also be used with static RDF datasets. Jelly was designed to be fast, well-compressed, and versatile.

Jelly can work with any RDF knowledge graph data, including RDF 1.1 and RDF 1.2.
Jelly can be used to represent streams of triples or quads and also graphs or datasets (RDF Messages).
Jelly can also be used to represent a single graph or dataset.
Jelly-SPARQL can be used to send SPARQL query results (SELECT and ASK), using the same compression as Jelly-RDF.
Jelly-Patch can be used to record changes to RDF datasets, including add/delete operations and transactions.
Jelly can be used for streaming data over the network (e.g., with MQTT, Kafka, gRPC), but also for working with flat files.
Jelly can compress RDF data on the fly, without having to know the data in advance.

Quick start

CLI tool

The easiest way to do something with Jelly is with the jelly-cli command line tool.

For Linux, macOS, and WSL on Windows, install it with the following command:

. <(curl -sSfL https://w3id.org/jelly/setup-cli.sh)
jelly-cli

You can also install jelly-cli manually by downloading the latest release, including a version for Windows without WSL and a platform-independent JAR file.

Run jelly-cli rdf to-jelly some-rdf-file.ttl > output.jelly to convert an RDF file to Jelly.
Run jelly-cli rdf from-jelly output.jelly to convert the Jelly file back to RDF.
Run jelly-cli sparql to-jelly results.srj > results.jellys to convert SPARQL query results to Jelly.
Run jelly-cli --help to see all available commands.

You can find more information about jelly-cli in its README on GitHub.

Example Jelly files

Go check out the Use cases page where we list links to example datasets in the Jelly format.

Apache Jena / RDF4J plugins

Check out the dedicated guide for installing plugins in Jena and RDF4J. You can use them to quickly add Jelly support to, for example, Apache Jena Fuseki and load Jelly files just like any other RDF file.

Neo4j plugin

We also have a plugin for Neo4j, extending the official neosemantics plugin. Check out installation instructions and usage guide.

Java & Scala programming

Go see the Jelly-JVM getting started guide for devs. It contains a lot of examples and code snippets for using Jelly in Java and Scala, with Jena, RDF4J, and Titanium.

Python programming

See the pyjelly getting started guide. It contains examples and code snippets for using Jelly with or without RDFLib, as well as other libraries like NetworkX.

Migrating from Jelly-RDF 1.1 to 1.2

Jelly-RDF 1.2 files can't be read by parsers that only support Jelly-RDF 1.1. The other way around works fine – parsers that support Jelly-RDF 1.2 can also read Jelly-RDF 1.1 and 1.0 files.

To convert your existing files to Jelly-RDF 1.2, use jelly-cli:

jelly-cli rdf to-jelly --in-format jelly old.jelly > new.jelly

If you need to send data to an older parser, you can also convert a Jelly-RDF 1.2 file back to 1.1:

jelly-cli rdf to-jelly --in-format jelly --opt.jelly-version=1.1 new.jelly > old.jelly

How does it work – encoding

Note

This guide describes the new Jelly-RDF 1.2 columnar encoding. To read about the Jelly-RDF 1.1 row-based encoding, see the old guide.

RDF data consists of triples (subject, predicate, object) or quads (subject, predicate, object, graph name). We will use these three triples in the N-Triples format as a running example:

<https://example.org/jelly> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> _:b1 .
<https://example.org/jelly> <https://example.org/label> "Jelly" .
_:b1 <https://example.org/label> "Serialization format"@en .

You can read that as "Jelly is an instance of a serialization format".

Jelly encodes a sequence of triples or quads as a binary stream, using efficient encoding. The first thing it does is splitting IRIs into prefixes and postfixes (we call postfixes "names"):

Prefixes:
- [1]: https://example.org/
- [2]: http://www.w3.org/1999/02/22-rdf-syntax-ns#
Names:
- [1]: jelly
- [2]: type
- [3]: label

They are communicated in the stream as part of a dynamically changing lookup table. If we run out of IDs, we remove old entries in the lookup to free up space. Each prefix and name is assigned a numerical ID starting with 1. We can now efficiently construct IRIs as a pair of IDs (1st ID is the prefix, the 2nd ID is the name):

https://example.org/jelly -> RdfIri(1, 1)
http://www.w3.org/1999/02/22-rdf-syntax-ns#type -> RdfIri(2, 2)
https://example.org/label -> RdfIri(1, 3)

This allows us to reuse repeating prefixes and names while saving on the serialized size. We also apply the same principle to datatype IRIs (they are not split in half, though). When we apply this across our 3 triples, we will get (in simplified form):

Subject	Predicate	Object
`RdfIri(1, 1)`	`RdfIri(2, 2)`	`"b1"`
`RdfIri(1, 1)`	`RdfIri(1, 3)`	`RdfLiteral("Jelly")`
`"b1"`	`RdfIri(1, 3)`	`RdfLiteral("Serialization format", "en")`

Jelly stores this table column by column – all subjects together, then all predicates, all objects, and all graph names. Within a column, the values are further split by type: IRIs go into one list, literals into another, blank nodes into yet another, and so on. If a column mixes types (like our objects column), Jelly adds 2 bits per value to say which list it comes from.

Notice that we have repeated IRIs in the first and second column. Jelly can compress that, by applying a method similar to Turtle's colons and semicolons, but a bit more general. If a column has the exact same term in consecutive triples/quads, we write the term only once:

Subject	Predicate	Object
`RdfIri(1, 1)`	`RdfIri(2, 2)`	`"b1"`
(empty)	`RdfIri(1, 3)`	`RdfLiteral("Jelly")`
`"b1"`	(empty)	`RdfLiteral("Serialization format", "en")`

Each column also has a short list of layout notes, which tell the parser where a term repeats and for how many rows, so that the parser can fill the repeated terms back in. One repeated term usually costs one one byte, and only a few bytes even if it repeats for thousands of rows, which makes this method very efficient.

Jelly also applies a delta compression scheme to prefix and name IDs. You can find the details in the spec, but the short version is that within each column we use 0 to indicate previous + 1 ID in the case of names, and previous ID in the case of prefixes. For our triples, this allows us to replace most name IDs with zeroes:

Subject	Predicate	Object
`RdfIri(1, 0)`	`RdfIri(2, 2)`	`"b1"`
(empty)	`RdfIri(1, 0)`	`RdfLiteral("Jelly")`
`"b1"`	(empty)	`RdfLiteral("Serialization format", "en")`

The value of 0 takes up just one byte, no matter how large the lookup is, allowing us to save even more space.

Jelly's encoding scheme was designed to work in a fully streaming manner. We can compress RDF data by only looking at one triple at a time, and we use a strictly limited amount of memory for that. This allows Jelly to work with datasets of any size – billions, trillions, and beyond.

That's the basics – see the spec for details.

Full translation of the example to Jelly

The running example was translated to a text-based readable version of Jelly with jelly-cli:

jelly-cli rdf to-jelly --opt.physical-type=triples | \
    jelly-cli rdf from-jelly --out-format jelly-text

Output:

# Frame 0
rows {
  options {
    physical_type: PHYSICAL_STREAM_TYPE_TRIPLES
    max_name_table_size: 4000
    max_prefix_table_size: 150
    max_datatype_table_size: 32
    version: 3
  }
}
columns {
  row_count: 3
  names {
    values: "jelly"
    values: "type"
    values: "label"
  }
  prefixes {
    values: "https://example.org/"
    values: "http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  }
  subjects {
    kinds: "\b"
    layouts: 0
    name_ids: 0
    prefix_ids: 1
    bnodes: "b1"
  }
  predicates {
    layouts: 32
    name_ids: 2
    name_ids: 0
    prefix_ids: 2
    prefix_ids: 1
  }
  objects {
    kinds: "\026"
    lex_values: "Jelly"
    lex_values: "Serialization format"
    literal_kinds: 0
    literal_kinds: 2
    langtags: "en"
    bnodes: "b1"
  }
}

What Jelly does and does not compress

Jelly is pretty good at compressing IRI-heavy RDF data, but it doesn't compress the text in the IRIs themselves. Same applies to blank node identifiers, language tags, and literal contents. If you care about small file size, you should compress your Jelly file with your compressor of choice, like gzip, bzip, or zstd. We found zstd to work particularly well in practice. While Jelly by itself can make datasets ~8x smaller than N-Triples (this largely depends on the dataset), with zstd compression, you can sometimes get it up to 100x.

How does it work – streams

A stream is just a sequence of items, possibly a very long one. We want to only look at a fixed number of elements of the stream at a time – this limits memory usage and allows us to process infinitely long streams.

The most basic kind of RDF stream is what you would see in an N-Triples or N-Quads file – after all, it's just a sequence of triples:

<https://example.org/jelly> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> _:b1 .
<https://example.org/jelly> <https://example.org/label> "Jelly" .
_:b1 <https://example.org/label> "Serialization format"@en .

We call this a flat RDF stream. Jelly supports this kind of streams – you will find them useful in transferring large RDF files, query responses, database dumps, and so on.

But, what if our stream contains a sequence of RDF graphs or datasets? For example, if we want to transfer the measurements of a temperature sensor, we would have:

Graph 1:

ex:sensor ex:measuredProperty ex:temperature ;
    ex:unit ex:Kelvin ;
    ex:value "280.4"^^xsd:decimal .

Graph 2:

ex:sensor ex:measuredProperty ex:temperature ;
    ex:unit ex:Kelvin ;
    ex:value "280.6"^^xsd:decimal .

...and so on. In traditional RDF systems, you had two options. You could modify the data, wrap it in containers, or apply some other processing to make it fit into one graph. Or you could simply put each graph/dataset in a separate file.

RDF Messages is a proposal by the RDF Stream Processing Community Group that offers a simpler way, fully implemented in Jelly. In this approach, multiple graphs or datasets can live within one file. We use messages as boundary markers:

# Message 1
ex:sensor ex:measuredProperty ex:temperature ;
    ex:unit ex:Kelvin ;
    ex:value "280.4"^^xsd:decimal .

# Message 2
ex:sensor ex:measuredProperty ex:temperature ;
    ex:unit ex:Kelvin ;
    ex:value "280.6"^^xsd:decimal .

We call this an RDF Message Stream. A Jelly parser can unpack this and process the messages one by one. The best part is that the compression (explained above) is applied across the entire stream. So, if an IRI appears in message 1, and then again in message 2, we will have to write it only once. This is very effective for data with repeating patterns, like IoT measurements, nanopublications, encyclopedic entries, or maps (cartography).

Note

Jelly predates the RDF Messages proposal, but it implements the same concept. Up to Jelly-RDF 1.1, each message was sent as one stream frame. Since Jelly-RDF 1.2, there is native support for RDF messages in the Jelly specification, and messages use dedicated delimiters. This allows for many messages in one stream frame, or for messages that span multiple stream frames.

As a summary:

Flat RDF stream – just a sequence of triples or quads. Great for processing a single file.
RDF Message Stream – a sequence of messages. Great if you have many small files.

Jelly records the stream type in the file header. Support for RDF Message Streams is optional in parsers. The physical layout of the stream is almost the same in both cases – a grouped stream just has extra delimiting markers. One important difference is that when using RDF Messages, the blank node identifiers are scoped to the message, so you can have the same blank node identifier in different messages without them being considered the same node.

The following sections contain more details about stream types in Jelly.

Streams and stream types in detail

Stream frames

Stream frames are batches of RDF statements (triples or quads), stored in columns. The frames have no meaning of their own – the producer can pick any frame size it likes.

Physical types (triples vs. quads)

The physical type tells you whether the stream contains triples or quads. There are two physical stream types in Jelly:

TRIPLES: data is encoded using triple statements. There is no information about the graph name in this type of stream.
QUADS: data is encoded using quad statements. Each quad has a graph name, which can also be the default graph.

Note

Jelly-RDF 1.1 also had the GRAPHS type, which was removed in Jelly-RDF 1.2. You can do the exact same thing with QUADS, which is simpler.

Stream types (flat vs. messages)

The stream type tells you whether the whole stream is one RDF graph or dataset (FLAT), or a sequence of RDF Messages (MESSAGES).

The Jelly-RDF 1.1 equivalent was logical stream types, taken directly from RDF-STaX.

Common patterns cookbook

Below you will find some common patterns for using Jelly. All of the presented patterns are supported in the Jelly-JVM implementation with the Reactive Streaming module.

Flat RDF triple stream – "just a bunch of triples"

Let's say you want to stream a lot of triples from A to B – maybe you're doing some kind of data migration, or you're sending data to a data lake. You don't care about the graph they belong to – you just want to send a bunch of triples.

This means you are using the FLAT stream type and the TRIPLES physical type, batching the triples into frames of an arbitrary size (let's say, 1000 triples each):

Example (click to expand)

Stream frame 1
- Stream options
- Triple 1
- Triple 2
- ...
- Triple 1000
Stream frame 2
- Triple 1001
- Triple 1002
- ...
- Triple 2000
...

You can then send these frames one-by-one over gRPC or Kafka, or write them to a file. The consumer will be able to read the triples one frame at a time, without having to know how many triples there are in total.

RDF graph stream

In this case we have (for example) an IoT sensor that periodically emits an RDF graph that describes what the sensor saw (something like SOSA/SSN). The graphs may be of different sizes (depending on what the sensor saw) and they can be emitted at different rates (depending on how often the sensor is triggered). We want to stream these graphs to a server that will process them in real-time with no additional latency.

This means you are using the MESSAGES stream type and the TRIPLES physical type, where the messages correspond to different unnamed (default) graphs. The simplest way is to put each graph in its own stream frame:

Example (click to expand)

Stream frame 1
- Stream options
- Triple 1 (of graph 1)
- Triple 2 (of graph 1)
- ...
- Triple 134 (of graph 1)
- End of message
Stream frame 2
- Triple 1 (of graph 2)
- Triple 2 (of graph 2)
- ...
- Triple 97 (of graph 2)
- End of message
...

The consumer will be able to read the graphs one message at a time, without having to know how many graphs there are in total.

RiverBench uses this pattern for distributing its triple streams (see example).

Flat RDF quad stream – "just a bunch of quads"

You want to stream a lot of quads – similar to the "just a bunch of triples" case above, but you also want to include the graph node. This means you are using the FLAT stream type and the QUADS physical type, batching the quads into frames of an arbitrary size (let's say, 1000 quads each):

Example (click to expand)

Stream frame 1
- Stream options
- Quad 1
- Quad 2
- ...
- Quad 1000
Stream frame 2
- Quad 1001
- Quad 1002
- ...
- Quad 2000
...

The mechanism is exactly the same as with a flat RDF triple stream.

RDF dataset stream

You want to stream RDF datasets – similar to the "a stream of graphs" case above, but your elements are entire datasets. This means you are using the MESSAGES stream type and the QUADS physical type, where the messages correspond to different datasets:

Example (click to expand)

Stream frame 1
- Stream options
- Quad 1 (of dataset 1)
- Quad 2 (of dataset 1)
- ...
- Quad 454 (of dataset 1)
- End of message
Stream frame 2
- Quad 1 (of dataset 2)
- Quad 2 (of dataset 2)
- ...
- Quad 323 (of dataset 2)
- End of message
...

The mechanism is exactly the same as with a triple stream of graphs.

RiverBench uses this pattern for distributing its RDF dataset streams (see example).

Ordering and delivery guarantees

To be able to compress RDF streams on-the-fly, Jelly requires that stream frames are kept strictly in order (see also the spec). This is because the compression algorithm updates its lookup tables dynamically over the course of the stream, and a given frame depends on the lookups defined in previous frames. If the frames are out of order, the compression may fail.

There are use cases where it's hard to guarantee strict ordering of messages, such as IoT messaging (e.g., MQTT with QoS 0) or high-throughput streams with parallel partitions (e.g., Kafka). In these cases you may want to employ one of these strategies:

Emit shared lookup tables at the start of the stream: If you know the vocabulary of the stream, you can emit most of the content of the lookup tables at the start of the stream, and then only update the lookup elements that vary frame-to-frame, keeping the updates local to the frame. This strategy is especially useful in IoT scenarios, where the vocabulary is usually known in advance. You don't need to modify the consumer in this case.
- A variation of this strategy is to communicate the lookup tables over a separate channel before starting the stream. This is useful if you can't guarantee that the lookup tables will be delivered before the stream frames.
Use a "frame ID" to keep track of the order: If you can't guarantee the order of the frames, you can add a "frame ID" to each frame, which will allow the consumer to reorder the frames before processing them. This strategy is useful in high-throughput scenarios, where you can't guarantee the order of the frames. You will need to modify the consumer to reorder the frames before processing them. However, handling failures in this scenario may be complicated.
Use partitions that are guaranteed to be in-order: If you can't guarantee the order of the frames, you can use partitions that are guaranteed to be in-order (e.g., Kafka partitions). Then, each partition should have its own set of lookups (essentially treating each partition as a separate stream in Jelly's terms). This strategy is useful in high-throughput scenarios.

Note that Jelly by default also assumes that frames are delivered at least once. At-least-once delivery is good enough (as long as the order is kept), as lookup updates are idempotent – you may only need to de-duplicate the frames afterwards. At-most-once delivery requires you to make the frames independent of each other, such as with the IoT strategy above.

Delimited vs. non-delimited Jelly

Protobuf messages by default are not delimited. This means that when you serialize a Protobuf message (e.g., a Jelly stream frame), the serialization does not include any information about where the message ends. This is fine when there is something else telling the parser where the message ends – for example, when you're sending the message over a gRPC, Kafka or MQTT stream – the streaming protocol tells the parser how long the message is. However, if you wanted to write multiple stream frames to a file, you would need to add some kind of delimiter between the frames – otherwise the parser would not know where one frame ends and the next one begins.

So, to summarize:

Use case	Jelly variant	Description
Jelly gRPC streaming protocol	Non-delimited	The gRPC protocol tells the parser how long the message is.
Streaming with Kafka, MQTT, or similar	Non-delimited	The streaming protocol tells the parser how long the message is.
Writing to a file	Delimited	You need to add a delimiter between the frames.
Writing to a raw network socket	Delimited	You need to add a delimiter between the frames.

The delimited variant works by adding an integer before the stream frame that specifies the length of the frame, in bytes. That's it.

You can read more about how this works in the serialization format specification.

Examples

Jelly-JVM supports both variants, but uses them in different contexts. When writing to a Java byte stream (typically a file) with Apache Jena RIOT or RDF4J Rio, the delimited variant is used. However, the RIOT/Rio integrations can parse either delimited or non-delimited Jelly data. In the gRPC protocol, the non-delimited variant is used.
RiverBench publishes its RDF metadata and datasets as Jelly files. These files are always written using the delimited variant.

Sample files

foaf-1.2.jelly is a simple Jelly file that may be useful when trying to understand how the format works. You can download it here:

foaf-1.2.jelly
foaf.ttl – same file in the Turtle format.
foaf-1.2.jelly.txt – text representation of the Jelly file, for debugging purposes.

The same file is also available in the older Jelly-RDF 1.0 format:

You can find more example datasets (including ones with millions of triples) on this page. The conformance test cases contain dozens of valid Jelly files testing all features of the format.

Implementing Jelly

Note

This section is intended only for those who want to write a new Jelly implementation from scratch. It's much easier to use an existing implementation, such as the JVM implementation.

Implementing Jelly from scratch is greatly simplified by the existing Protobuf and RDF libraries. Essentially, the only thing you'll need to do is to glue them together:

Find a Protobuf library for your language. You can find a list of official Protobuf implementations here and a list of community-maintained implementations here.
Use the library to generate the code for the Jelly messages (this usually involves using protoc). You can find the Protobuf definitions in the jelly-protobuf repository.
Find an RDF library for your language. You can find a list of RDF libraries here.
Implement conversions to and/or from the RDF library's data structures. You can find an example of the conversion code in the Jelly-JVM implementation (core, jena, and rdf4j modules).
In the implementation follow the specification to ensure compatibility.

That's it! You may also want to implement streaming facilities, such as Reactive Streams in Java/Scala. Implementing the gRPC publish/subscribe mechanism follows a very similar procedure – many Protobuf libraries have built-in support for gRPC with code generation.

Jelly user guide

What can it do?

Quick start

CLI tool

Apache Jena / RDF4J plugins

Neo4j plugin

Java & Scala programming

Python programming

Migrating from Jelly-RDF 1.1 to 1.2

How does it work – encoding

What Jelly does and does not compress

How does it work – streams

Streams and stream types in detail

Stream frames

Physical types (triples vs. quads)

Stream types (flat vs. messages)

Common patterns cookbook

Flat RDF triple stream – "just a bunch of triples"

RDF graph stream

Flat RDF quad stream – "just a bunch of quads"

RDF dataset stream

Ordering and delivery guarantees

Delimited vs. non-delimited Jelly

Examples

Sample files

Implementing Jelly

See also