Dat awesome

A curated list of the Dat Project ecosystem.

Please read the contribution guidelines before contributing.

Want to learn more? Check out:

Dat Applications

User facing applications for sharing, downloading, and managing dats.

dat - command line interface for managing dat archives
Dat Desktop - desktop app for managing dat archives
Beaker Browser - a peer-to-peer browser with tools to create and host websites.

Community Applications

Projects built using Dat to share and transfer data. Open a PR to add your project here!

sciencefair - The open source p2p desktop science library that puts users in control 🔬 📖 http://sciencefair-app.com
hyperirc - bot that mirrors irc channels to a hypercore read-only log
soundcloud-archiver - decentralized archives rescue music in case soundcloud shuts down
hypervision - watch and broadcast peer-to-peer live video streams
hypertweet - stream your twitter feed to a hypercore feed
dat-photos-app - decentralized, peer-to-peer photo sharing app for beaker browser

Using Dat

Modules that help you build things on top of Dat:

High-Level APIs

High-level APIs that act as glue for many of the Dat modules:

dat-node - Node module for creating Dat applications with distributed file systems.
dat-js - A pure JavaScript browser-friendly api for using dat over webrtc
pauls-dat-api - Library of functions that make working with dat / hyperdrive easier.
node-dat-archive - node api that supports beaker browser DatArchive format, uses pauls-dat-api

Hosting & Dat Management

Tools for hosting Dats, managing sets of Dats, etc.

hypercore-archiver - archiver peer that backs up multiple hypercore / hyperdrive feeds to disk
hypercloud - p2p + ☁
hashbase - hosting for the peer-to-peer web
dat-now - publish live syncing and versioned websites, files or whatever to now.sh instantly

Utilities for hypercore-archiver:

hypercore-archiver-bot - IRC bot that provides an interface to hypercore-archiver
hypercore-archiver-ws - websocket server for hypercore-archiver

Dat project runs a registry at datproject.org. We use these tools to manage our dats:

dat-registry-api - account registry api for dat archives with user accounts, uses township
dat-registry-client - client for registry api for user registration, login and publishing

Managing & Aggregating Dats

multidat - manage dat archives in multiple locations, uses a dat factory, based on multidrive
multidrive - manage multiple hyperdrive archives located anywhere on the filesystem
dat-pki - A public key infrastructure with many encryption utilities for Dat filesharing
injestdb - decentralized table/records-based database with sql-like behaviour, uses node-dat-archive

Http Hosting

hyperdrive-http - serve hyperdrive archives over http
dathttpd - A Web server for Dat and HTTPS, with zero-config TLS.

Dat Link Utilties

Resolving, parsing, encoding dat links.

dat-dns - issue dns lookups for dat archives using https requests to a target host
dat-link-resolve - resolve dat url's, links to a dat key using common methods, uses dat-dns
parse-dat-url - node's url.parse updated to support versioned dat url's
dat-encoding - encoder and decoder that supports the dat url-scheme

Dat Utilities

Utilities to show information about an existing dat archive.

dat-log - simple cli that lists the history of a dat archive
dat-ls - simple cli that lists all the changes in a dat archive
hyperhealth - monitor health of hyperdrive or dat archives, e.g. peer count and peer mirror %
hyperdrive-network-speed - track upload and download speeds on hyperdrive archives

File Imports & Exports

hyperdrive-import-files - import files and folders into a hyperdrive archive
mirror-folder - Small module to mirror a folder to another folder. Supports live mode as well.
hyperdrive-staging-area - staging area for uncommited writes that can sync to a hyperdrive archive
hyperdrive-to-zip-stream - export hyperdrive archives as a zip files

Hypercore Tools

Tools for using hypercore feeds

hyperpipe - simple cli to pipe and read files into live hypercore feeds

Dat Core Modules

Things we used to build Dat. Dat tools (CLI, Desktop, dat-node) are opinonated versions of hyperdrive that work well for using facing applications.

hyperdrive - secure, decentralized peer-to-peer file system on top of hypercore
hypercore - decentralized peer-to-peer append-only logs using hypercore protocol

CLI Utilities

Utilities used in our command line interface

dat-doctor - diagnose networking problems for dat, comes bundled with dat-cli
dat-ignore - check files against .datignore before adding to a dat archive
dat-json - read and write dat.json files, uses toiletdb

Networking

hyperdiscovery - join the p2p swarm for hypercore feeds, uses discovery-swarm
discovery-swarm - discover and connect to peers, uses discovery-channel
webrtc-swarm - create a swarm of p2p connections using webrtc and a signalhub
dat-swarm-defaults - default configuration for dns and dht for use with discovery-swarm

Lower level networking modules

discovery-channel - search discovery networks to find answering peers
dns-discovery - discover peers using regular- and multicast-dns
multicast-dns - low-level multicast-dns implementation in pure javascript
bittorrent-dht - complete js implementation of DHT peer discovery protocol
utp-native - utp protocol implementation, based on libutp native bindings
signalhub - simple signalling server that can be used to coordinate handshaking with webrtc

Storage

dat-storage - Dat specific storage provider for Hyperdrive
dat-secret-storage - hyperdrive storage module for dat secret keys

Random Access

Dat relies on random access storage. Any of these modules can be used to provide the storage for a dat archive.

abstract-random-access - base class for random access stores
multi-random-access - combine multiple abstract-random-access stores into a single one

The following are specific implementations of abstract-random-access:

random-access-file - continuous reading / writing to files using random offset and lengths
random-access-memory - same as random-access-file but maintains data in memory
random-access-page-files - backend writing to fixed size page files, useful for sparse data
dat-http - http transport and storage provider for dat archives
random-access-idb - random-access-compatible indexedDB storage layer

More modules from Dat project that are related to Dat but may not be used currently.

peer-network - create internet-accessible servers/clients listening on names, not hostnames
hyperdht - DHT that supports peer discovery and distributed hole punching

Dat Project Organization Stuff

datproject.org - datproject.org on github
discussions - general dat project discussion, ideas, feature requests
styleguide - styleguide and visual assets for the dat project
dat-elements - reusable ui elements for dat-based apps, such as loader, sprite, icon
dat-colors - css color definitions that match dat styleguide
dat-icons - svg icon definitions that match dat styleguide
dat.json specification - specification of the dat.json metadata format

Outdated

Modules that are currently outdated. We released a major breaking change (hypercore v6, hyperdrive v9) in May 2017. This change included major speed and storage improvements. These modules need to be updated to support the new API.

If you want to update one of these, we are happy to help you choose one depending on your interests and what may still be useful.

dat.haus - dat + http + unix, the composable http api to the dat network
hyperfeed - publish decentralized rss, atom or rdf feeds, based on hyperdrive and feed
normcore - no-config distributed streams using hypercore
github-to-hypercore - stream github event feeds into hypercore feeds, uses normcore
hyperspark - decentralized data processing platform for dat archives, inspired by spark
hypercore-index - linear asynchronous stateful indexing of a hypercore feed
hyperdrive-protocol - message encoding used by hyperdrive
hyperdrive-http-server - small cli to serve hyperdrive archives over http
dat-publish - small cli to publish dat archives to a hyperdrive-http or dat-archiver server
dat-push - small cli for pushing files to a dat-archiver or dat-publish server
dat-backup - backup a dat archive as a single file to local storage and retain full history
archiver-server - serve hypercore-archiver feeds over the dat network and http, uses discovery-swarm
archiver-api - simple rest api for accessing hypercore-archiver peers
hyperdrive-ln - create symbolic links between hyperdrive archives
hyperdrive-multiwriter - present a bundle of hyperdrive archives together as a multi-writer view
hyperdrive-named-archives - create hyperdrive archives that store and load link keys from names
git-dat - git plugin to use dat archives as remotes for a git repository
jawn - distributed version control for tabular data, based on hypercore
dat-archiver - archiver peer that backs up dat archives, based on hypercore-archiver
hyperdrive-stats - live and persistent statistics tracker for hyperdrive archives
hyperdrive-stats-server - server for sending hypercore / hyperdrive stats over server-side events
hyperdrive-stats-ui - html-based user interface to hypercore-stats-server
zip-to-hyperdrive - import contents of a zip archive into a hyperdrive archive
url-dat - import files from http url's into a hyperdrive archive, uses tar-dat
tar-dat - stream tar files into a hyperdrive archive
hyperdrive-duplicate - check if a file is a duplicate to a hyperdrive archive entry

License

To the extent possible under law, the Dat team, and contributors have waived all copyright and related or neighboring rights to this work.