earthstar API tour

Table of contents

• Introduction
• Peer
• Replica
• Replica drivers
• Validators
• Identity
• Query
• Doc
• Crypto
• setGlobalCryptoDriver
• Syncers
• QueryFollower
• ReplicaCache

Introduction

earthstar is a reference implementation of Earthstar written in Typescript. You can use it to add Earthstar functionality to programs running on servers, browsers, or the command line.

It can be imported via URL into a browser:

<script type="module">
  import * as Earthstar from "https://cdn.earthstar-project.org/js/earthstar.bundle.v9.3.3.js";
</script>

Or Deno:

import * as Earthstar from "https://deno.land/x/earthstar@v9.3.3/mod.ts";

or installed with NPM:

npm install earthstar@9.3.3

And import what you need for your specific platform:

import { Replica } from "earthstar";
import { ReplicaDriverIndexedDB } from "earthstar/browser";
import { ReplicaDriverSqlite } from "earthstar/node";

To integrate Earthstar into your program, your application will probably need a single instance of Earthstar's Peer class, which holds the replicas for many shares and is able to synchronise them with other peers.

Nearly all of Earthstar's classes can be configured to suit your specific use-case, For example, a Replica can be configured to persist data to IndexedDB if you're building a browser app, or Sqlite if you're building a server or command-line tool.

Peer

A Peer is responsible for holding many Replicas, and offers many ways to access and manage them. A Peer can hold one Replica for each share.

There is also a Peer.sync method which can be passed 'syncable' things such as another Peer or a Websocket or HTTP URL to a replica server. The Peer will automatically create a new syncer

const myPeer = new Peer();

// Add a replica
myPeer.addReplica(myReplica);

// Sync with a local instance of a `Peer`
myPeer.sync(myOtherPeer);
// Sync with a replica server using Websockets
myPeer.sync("wss://our.club/earthstar");

// ... Some time later...

myPeer.stopSyncing();

When two peers synchronise with one another, they perform a 'salted handshake' to determine if they have any shares in common without actually revealing what those shares are. Whichever shares they have in common are synced.

Replica

A local replica of a share's documents. Provides operations for querying and setting documents.

import { ReplicaAsync, ValidatorEs4, ReplicaDriverMemory } from "earthstar";

// Construct a replica
const myReplica = new Replica(
  // Needs a share address,
  `+myshare.a8puorlpz1`,
  // A document validator,
  ValidatorEs4,
  // And a driver to teach it how to read and write documents
  new ReplicaDriverMemory(`+myshare.a8puorlpz1`)
);

// Query for documents
const txtResults = myReplica.query({
  filter: {
    pathEndsWith: ".txt"
  }
});

// Set a document
await myReplica.set(myKeypair, {
  format: "es.4",
  path: "notes/ink.txt",
  content: "Don't forget to get some new ink."
});

Replica drivers

Earthstar can be used on in browsers, on servers, the command line, or anywhere JavaScript can be run. There's no single way to persist data in all these environments, which is why we made Replica able to use different ReplicaDrivers suited to the environment its being used in.

Universal drivers:

• ReplicaDriverMemory (no persistence)

Web platform drivers:

• ReplicaDriverLocalReplica
• ReplicaDriverIndexedDB

Deno drivers:

• ReplicaDriverSqlite

Node drivers:

• ReplicaDriverSqlite

It's possible to create your own replica drivers which conform to IReplicaDriver. Technical docs for IReplicaDriver

Validators

Earthstar shares are able to store documents of different formats. But most of the time, you'll want to use the built-in one from this library: es.4. As such, Earthstar currenly only offers one format validator: FormatValidatorEs4.

Every time a new document is ingested by a ReplicaAsync it is checked by a validator. In the case of FormatValidatorEs4, it checks whether a document's signature matches its author, space address, and content.

It's possible to write your own validator which adheres to IFormatValidator. Technical docs for IFormatValidator

Identity

An identity is a keypair containing a public address and a secret. This keypair is used to sign documents whenever they're written. The public address can be freely distributed, but the secret should be, well, secret.

You can generate identities using Crypto.

Note that documents are not encrypted (yet), only signed. This means we can verify that the document has not been tampered with on its journey to you, making it possible for syncing to hop across several peers without worrying about them messing with your data. They can still read your data though -- all docs in a space are readable by everyone in the space.

Technically speaking, these are ed25519 keypairs encoded in base32.

Query

Used to describe a document query. Used with ReplicaAsync.query and QueryFollower.

Doc

Represents some cryptographically-signed user data. The contents of a Doc can only be changed via ReplicaAsync. A document cannot be moved from one share to another.

Crypto

Crypto is used internally for signing and verifying documents using ed25519. In most cases, you will want to use it to generate new identity keypairs:

const keypair = Crypto.generateAuthorKeypair("suzy");

console.log(keypair);
/*
  {
    "address":"@suzy.bv3cclqhvnbn4fwjur7u4nimm4cznbzigqyihlfwi6acck5b4q54q",
    "secret":"bxrishindifrczhm5riqd4mozx4o5htksqthd64yph4rlfs6ej53n"
  }
*/

setGlobalCryptoDriver

Because ed25519 is not part of any JS standard library (or WebCrypto API), there are several third-party implementations, some of which are only available for certain runtimes.

To balance compatibility and performance, Earthstar uses noble/ed25519 by default. This works in browsers, Deno, and Node.

Earthstar exports a setGlobalCrypto function to change this:

import { setGlobalCryptoDriver, CryptoDriverChloride } from "earthstar";

setGlobalCryptoDriver(CryptoDriverChloride);

Syncers

Most of the time you will want to use the Peer.sync convenience method to synchronise with other peers. But if you are building something special — for example a server, or something with Web Workers and BroadcastChannel — Syncer provides a more flexible API which you can use in combination with classes conforming to earthstar-streaming-rpc's ITransport type.

import * as Rpc from "https://deno.land/x/earthstar_streaming_rpc/mod.ts";

const myPeer = new Peer();

// This syncer will listen for other syncers using the `my_channel` BroadcastChannel.
const broadcastChannelSyncer = new Syncer(
  myPeer,
  (methods) => new Rpc.TransportBroadcastChannel({
    methods,
    deviceId: myPeer.peerId,
    channel: 'my_channel'
  })
);

// Don't forget to close once done!
broadcastChannelSyncer.close();

QueryFollower

Listens to events from a Replica, and if it matches the given query, sends an event to a SuperBus.

Here's a basic usage:

const myMarkdownFollower = new QueryFollower(myReplica, {
  filter: {
    pathEndsWith: ".md",
  },
});

// Print every newly created / updated Markdown doc to the console
myMarkdownFollower.bus.on((event: LiveQueryEvent) => {
  if (event.kind === "success") {
    console.log("A markdown doc was updated or created!");
  }
});

// Start the follower.
myMarkdownFollower.hatch();

A QueryFollower must always be hatched first. Hatching makes it catch up with all existing documents which match the query, after which it starts listening for new events.

When you're done with a QueryFollower, you should close it to stop callbacks from running:

myMarkdownFollower.close();

ReplicaCache

ReplicaAsync offers a strictly asynchronous interface to a share's contents. Sometimes a synchronous API would make life far easier, and ReplicaCache does its best to provide that.

ReplicaCache offers a synchronous API for fetching documents by storing queried documents in an internal cache.

When you first query a document, there is no result. But ReplicaCache can be subscribed to for updates, at which point you return to the cache and get an instant result:

import { ReplicaCache } from "earthstar";

const cache = new ReplicaCache(myReplica);

const myQuery = { filter: { pathStartsWith: "/about/" } };

let aboutDocs = cache.queryDocs(myQuery); // This will be an empty list.

cache.onCacheUpdated(() => {
  // Trigger a re-querying of the cache!
  aboutDocs = cache.queryDocs(myQuery); // Now there should be something in there.
});

ReplicaCache has an optional TTL (time-to-live) option which can be provided to its constructor:

const cache = new ReplicaCache(myReplica, 2000); // ms, 1000 by default

When you fetch an expired result from the cache, ReplicaCache will then return the latest result from its backing replica.