@@ -1,14 +1,96 @@ | |||||
About | |||||
===== | |||||
Matrix is an ambitious new ecosystem for open federated Instant Messaging and VoIP[1]. | |||||
Matrix specifies a set of pragmatic RESTful HTTP JSON APIs as an open standard, providing: | |||||
- Creating and managing fully distributed chat rooms with no | |||||
single points of control or failure | |||||
- Eventually-consistent cryptographically secure synchronisation of room | |||||
state across a global open network of federated servers and services | |||||
- Sending and receiving extensible messages in a room with (optional) | |||||
end-to-end encryption[2] | |||||
- Inviting, joining, leaving, kicking, banning room members | |||||
- Managing user accounts (registration, login, logout) | |||||
- Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers, | |||||
Facebook accounts to authenticate, identify and discover users on Matrix. | |||||
- Placing 1:1 VoIP and Video calls (in development) | |||||
These APIs are intended to be implemented on a wide range of servers, services | |||||
and clients which then form the Matrix ecosystem, and allow developers to build | |||||
messaging and VoIP functionality on top of the open Matrix community rather than | |||||
using closed or proprietary solutions. The hope is for Matrix to act as the | |||||
building blocks for a new generation of fully open and interoperable messaging | |||||
and VoIP apps for the internet. | |||||
Synapse is a reference "homeserver" implementation of Matrix from the core | |||||
development team at matrix.org, written in Python/Twisted for clarity and | |||||
simplicity. It is intended to showcase the concept of Matrix and let folks see | |||||
the spec in the context of a codebase and let you run your own homeserver and | |||||
generally help bootstrap the ecosystem. | |||||
In Matrix, every user runs one or more Matrix clients, which connect through to | |||||
a Matrix homeserver which stores all their personal chat history and user | |||||
account information - much as a mail client connects through to an IMAP/SMTP | |||||
server. Just like email, you can either run your own Matrix homeserver and | |||||
control and own your own communications and history or use one hosted by someone | |||||
else (e.g. matrix.org) - there is no single point of control or mandatory | |||||
service provider in Matrix, unlike WhatsApp, Facebook, Hangouts, etc. | |||||
Synapse ships with two basic demo Matrix clients: webclient (a basic group chat web client demo implemented in AngularJS) and cmdclient (a basic Python commandline utility which lets you easily see what the JSON APIs are up to). | |||||
We'd like to invite you to take a look at the Matrix spec, try to run a homeserver, and join the existing Matrix chatrooms already out there, experiment with the APIs and the demo clients, and let us know your thoughts at https://github.com/matrix-org/synapse/issues or at matrix@matrix.org. | |||||
Thanks for trying Matrix! | |||||
[1] VoIP currently in development | |||||
[2] End-to-end encryption is currently in development | |||||
Directory Structure | |||||
=================== | |||||
:: | |||||
. | |||||
├── cmdclient Basic CLI python Matrix client | |||||
├── demo Scripts for running standalone Matrix demos | |||||
├── docs All doc, including the draft Matrix API spec | |||||
│ ├── client-server The client-server Matrix API spec | |||||
│ ├── model Domain-specific elements of the Matrix API spec | |||||
│ ├── server-server The server-server model of the Matrix API spec | |||||
│ └── sphinx The internal API doc of the Synapse homeserver | |||||
├── experiments Early experiments of using Synapse's internal APIs | |||||
├── graph Visualisation of Matrix's distributed message store | |||||
├── synapse The reference Matrix homeserver implementation | |||||
│ ├── api Common building blocks for the APIs | |||||
│ │ ├── events Definition of state representation Events | |||||
│ │ └── streams Definition of streamable Event objects | |||||
│ ├── app The __main__ entry point for the homeserver | |||||
│ ├── crypto The PKI client/server used for secure federation | |||||
│ │ └── resource PKI helper objects (e.g. keys) | |||||
│ ├── federation Server-server state replication logic | |||||
│ ├── handlers The main business logic of the homeserver | |||||
│ ├── http Wrappers around Twisted's HTTP server & client | |||||
│ ├── rest Servlet-style RESTful API | |||||
│ ├── storage Persistence subsystem (currently only sqlite3) | |||||
│ │ └── schema sqlite persistence schema | |||||
│ └── util Synapse-specific utilities | |||||
├── tests Unit tests for the Synapse homeserver | |||||
└── webclient Basic AngularJS Matrix web client | |||||
Installation | Installation | ||||
============ | ============ | ||||
[TODO(kegan): I also needed libffi-dev, which I don't think is included in build-essential.] | |||||
First, the dependencies need to be installed. Start by installing | |||||
'python2.7-dev' and the various tools of the compiler toolchain. | |||||
First, the dependencies need to be installed. Start by installing 'python-dev' | |||||
and the various tools of the compiler toolchain: | |||||
N.B. that python 2.x where x >= 7 is required. | |||||
Installing prerequisites on ubuntu:: | Installing prerequisites on ubuntu:: | ||||
$ sudo apt-get install build-essential python-dev | |||||
$ sudo apt-get install build-essential python2.7-dev libffi-dev | |||||
Installing prerequisites on Mac OS X:: | Installing prerequisites on Mac OS X:: | ||||
@@ -35,12 +117,13 @@ This should end with a 'PASSED' result:: | |||||
PASSED (successes=143) | PASSED (successes=143) | ||||
Running The Home Server | |||||
======================= | |||||
Running The Synapse Homeserver | |||||
============================== | |||||
In order for other home servers to send messages to your server, they will need | |||||
to know its host name. You have two choices here, which will influence the form | |||||
of your user IDs: | |||||
In order for other homeservers to send messages to your server, it will need to | |||||
be publicly visible on the internet, and they will need to know its host name. | |||||
You have two choices here, which will influence the form of your matrix user | |||||
IDs: | |||||
1) Use the machine's own hostname as available on public DNS in the form of its | 1) Use the machine's own hostname as available on public DNS in the form of its | ||||
A or AAAA records. This is easier to set up initially, perhaps for testing, | A or AAAA records. This is easier to set up initially, perhaps for testing, | ||||
@@ -58,10 +141,9 @@ For the first form, simply pass the required hostname (of the machine) as the | |||||
For the second form, first create your SRV record and publish it in DNS. This | For the second form, first create your SRV record and publish it in DNS. This | ||||
needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname | needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname | ||||
and port where the server is running. (At the current time we only support a | |||||
single server, but we may at some future point support multiple servers, for | |||||
backup failover or load-balancing purposes). The DNS record would then look | |||||
something like:: | |||||
and port where the server is running. (At the current time synapse does not | |||||
support clustering multiple servers into a single logical homeserver). The DNS | |||||
record would then look something like:: | |||||
_matrix._tcp IN SRV 10 0 8448 machine.my.domain.name. | _matrix._tcp IN SRV 10 0 8448 machine.my.domain.name. | ||||
@@ -73,9 +155,13 @@ SRV record, as that is the name other machines will expect it to have:: | |||||
You may additionally want to pass one or more "-v" options, in order to | You may additionally want to pass one or more "-v" options, in order to | ||||
increase the verbosity of logging output; at least for initial testing. | increase the verbosity of logging output; at least for initial testing. | ||||
For the initial alpha release, the homeserver is not speaking TLS for | |||||
either client-server or server-server traffic for ease of debugging. We have | |||||
also not spent any time yet getting the homeserver to run behind loadbalancers. | |||||
Running The Web Client | |||||
====================== | |||||
Running The Demo Web Client | |||||
=========================== | |||||
At the present time, the web client is not directly served by the homeserver's | At the present time, the web client is not directly served by the homeserver's | ||||
HTTP server. To serve this in a form the web browser can reach, arrange for the | HTTP server. To serve this in a form the web browser can reach, arrange for the | ||||
@@ -92,6 +178,7 @@ HTML5 local storage to remember its config), you will need to log in to your | |||||
account. If you don't yet have an account, because you've just started the | account. If you don't yet have an account, because you've just started the | ||||
homeserver for the first time, then you'll need to register one. | homeserver for the first time, then you'll need to register one. | ||||
Registering A New Account | Registering A New Account | ||||
------------------------- | ------------------------- | ||||
@@ -103,7 +190,10 @@ account. Your name will take the form of:: | |||||
(pronounced "at localpart on my dot domain dot here") | (pronounced "at localpart on my dot domain dot here") | ||||
Specify your desired localpart in the topmost box of the "Register for an | Specify your desired localpart in the topmost box of the "Register for an | ||||
account" form, and click the "Register" button. | |||||
account" form, and click the "Register" button. Hostnames can contain ports if | |||||
required due to lack of SRV records (e.g. @matthew:localhost:8080 on an internal | |||||
synapse sandbox running on localhost) | |||||
Logging In To An Existing Account | Logging In To An Existing Account | ||||
--------------------------------- | --------------------------------- | ||||
@@ -0,0 +1,154 @@ | |||||
import curses | |||||
import curses.wrapper | |||||
from curses.ascii import isprint | |||||
from twisted.internet import reactor | |||||
class CursesStdIO(): | |||||
def __init__(self, stdscr, callback=None): | |||||
self.statusText = "Synapse test app -" | |||||
self.searchText = '' | |||||
self.stdscr = stdscr | |||||
self.logLine = '' | |||||
self.callback = callback | |||||
self._setup() | |||||
def _setup(self): | |||||
self.stdscr.nodelay(1) # Make non blocking | |||||
self.rows, self.cols = self.stdscr.getmaxyx() | |||||
self.lines = [] | |||||
curses.use_default_colors() | |||||
self.paintStatus(self.statusText) | |||||
self.stdscr.refresh() | |||||
def set_callback(self, callback): | |||||
self.callback = callback | |||||
def fileno(self): | |||||
""" We want to select on FD 0 """ | |||||
return 0 | |||||
def connectionLost(self, reason): | |||||
self.close() | |||||
def print_line(self, text): | |||||
""" add a line to the internal list of lines""" | |||||
self.lines.append(text) | |||||
self.redraw() | |||||
def print_log(self, text): | |||||
self.logLine = text | |||||
self.redraw() | |||||
def redraw(self): | |||||
""" method for redisplaying lines | |||||
based on internal list of lines """ | |||||
self.stdscr.clear() | |||||
self.paintStatus(self.statusText) | |||||
i = 0 | |||||
index = len(self.lines) - 1 | |||||
while i < (self.rows - 3) and index >= 0: | |||||
self.stdscr.addstr(self.rows - 3 - i, 0, self.lines[index], | |||||
curses.A_NORMAL) | |||||
i = i + 1 | |||||
index = index - 1 | |||||
self.printLogLine(self.logLine) | |||||
self.stdscr.refresh() | |||||
def paintStatus(self, text): | |||||
if len(text) > self.cols: | |||||
raise RuntimeError("TextTooLongError") | |||||
self.stdscr.addstr( | |||||
self.rows - 2, 0, | |||||
text + ' ' * (self.cols - len(text)), | |||||
curses.A_STANDOUT) | |||||
def printLogLine(self, text): | |||||
self.stdscr.addstr( | |||||
0, 0, | |||||
text + ' ' * (self.cols - len(text)), | |||||
curses.A_STANDOUT) | |||||
def doRead(self): | |||||
""" Input is ready! """ | |||||
curses.noecho() | |||||
c = self.stdscr.getch() # read a character | |||||
if c == curses.KEY_BACKSPACE: | |||||
self.searchText = self.searchText[:-1] | |||||
elif c == curses.KEY_ENTER or c == 10: | |||||
text = self.searchText | |||||
self.searchText = '' | |||||
self.print_line(">> %s" % text) | |||||
try: | |||||
if self.callback: | |||||
self.callback.on_line(text) | |||||
except Exception as e: | |||||
self.print_line(str(e)) | |||||
self.stdscr.refresh() | |||||
elif isprint(c): | |||||
if len(self.searchText) == self.cols - 2: | |||||
return | |||||
self.searchText = self.searchText + chr(c) | |||||
self.stdscr.addstr(self.rows - 1, 0, | |||||
self.searchText + (' ' * ( | |||||
self.cols - len(self.searchText) - 2))) | |||||
self.paintStatus(self.statusText + ' %d' % len(self.searchText)) | |||||
self.stdscr.move(self.rows - 1, len(self.searchText)) | |||||
self.stdscr.refresh() | |||||
def logPrefix(self): | |||||
return "CursesStdIO" | |||||
def close(self): | |||||
""" clean up """ | |||||
curses.nocbreak() | |||||
self.stdscr.keypad(0) | |||||
curses.echo() | |||||
curses.endwin() | |||||
class Callback(object): | |||||
def __init__(self, stdio): | |||||
self.stdio = stdio | |||||
def on_line(self, text): | |||||
self.stdio.print_line(text) | |||||
def main(stdscr): | |||||
screen = CursesStdIO(stdscr) # create Screen object | |||||
callback = Callback(screen) | |||||
screen.set_callback(callback) | |||||
stdscr.refresh() | |||||
reactor.addReader(screen) | |||||
reactor.run() | |||||
screen.close() | |||||
if __name__ == '__main__': | |||||
curses.wrapper(main) |
@@ -0,0 +1,380 @@ | |||||
# -*- coding: utf-8 -*- | |||||
""" This is an example of using the server to server implementation to do a | |||||
basic chat style thing. It accepts commands from stdin and outputs to stdout. | |||||
It assumes that ucids are of the form <user>@<domain>, and uses <domain> as | |||||
the address of the remote home server to hit. | |||||
Usage: | |||||
python test_messaging.py <port> | |||||
Currently assumes the local address is localhost:<port> | |||||
""" | |||||
from synapse.federation import ( | |||||
ReplicationHandler | |||||
) | |||||
from synapse.federation.units import Pdu | |||||
from synapse.util import origin_from_ucid | |||||
from synapse.app.homeserver import SynapseHomeServer | |||||
#from synapse.util.logutils import log_function | |||||
from twisted.internet import reactor, defer | |||||
from twisted.python import log | |||||
import argparse | |||||
import json | |||||
import logging | |||||
import os | |||||
import re | |||||
import cursesio | |||||
import curses.wrapper | |||||
logger = logging.getLogger("example") | |||||
def excpetion_errback(failure): | |||||
logging.exception(failure) | |||||
class InputOutput(object): | |||||
""" This is responsible for basic I/O so that a user can interact with | |||||
the example app. | |||||
""" | |||||
def __init__(self, screen, user): | |||||
self.screen = screen | |||||
self.user = user | |||||
def set_home_server(self, server): | |||||
self.server = server | |||||
def on_line(self, line): | |||||
""" This is where we process commands. | |||||
""" | |||||
try: | |||||
m = re.match("^join (\S+)$", line) | |||||
if m: | |||||
# The `sender` wants to join a room. | |||||
room_name, = m.groups() | |||||
self.print_line("%s joining %s" % (self.user, room_name)) | |||||
self.server.join_room(room_name, self.user, self.user) | |||||
#self.print_line("OK.") | |||||
return | |||||
m = re.match("^invite (\S+) (\S+)$", line) | |||||
if m: | |||||
# `sender` wants to invite someone to a room | |||||
room_name, invitee = m.groups() | |||||
self.print_line("%s invited to %s" % (invitee, room_name)) | |||||
self.server.invite_to_room(room_name, self.user, invitee) | |||||
#self.print_line("OK.") | |||||
return | |||||
m = re.match("^send (\S+) (.*)$", line) | |||||
if m: | |||||
# `sender` wants to message a room | |||||
room_name, body = m.groups() | |||||
self.print_line("%s send to %s" % (self.user, room_name)) | |||||
self.server.send_message(room_name, self.user, body) | |||||
#self.print_line("OK.") | |||||
return | |||||
m = re.match("^paginate (\S+)$", line) | |||||
if m: | |||||
# we want to paginate a room | |||||
room_name, = m.groups() | |||||
self.print_line("paginate %s" % room_name) | |||||
self.server.paginate(room_name) | |||||
return | |||||
self.print_line("Unrecognized command") | |||||
except Exception as e: | |||||
logger.exception(e) | |||||
def print_line(self, text): | |||||
self.screen.print_line(text) | |||||
def print_log(self, text): | |||||
self.screen.print_log(text) | |||||
class IOLoggerHandler(logging.Handler): | |||||
def __init__(self, io): | |||||
logging.Handler.__init__(self) | |||||
self.io = io | |||||
def emit(self, record): | |||||
if record.levelno < logging.WARN: | |||||
return | |||||
msg = self.format(record) | |||||
self.io.print_log(msg) | |||||
class Room(object): | |||||
""" Used to store (in memory) the current membership state of a room, and | |||||
which home servers we should send PDUs associated with the room to. | |||||
""" | |||||
def __init__(self, room_name): | |||||
self.room_name = room_name | |||||
self.invited = set() | |||||
self.participants = set() | |||||
self.servers = set() | |||||
self.oldest_server = None | |||||
self.have_got_metadata = False | |||||
def add_participant(self, participant): | |||||
""" Someone has joined the room | |||||
""" | |||||
self.participants.add(participant) | |||||
self.invited.discard(participant) | |||||
server = origin_from_ucid(participant) | |||||
self.servers.add(server) | |||||
if not self.oldest_server: | |||||
self.oldest_server = server | |||||
def add_invited(self, invitee): | |||||
""" Someone has been invited to the room | |||||
""" | |||||
self.invited.add(invitee) | |||||
self.servers.add(origin_from_ucid(invitee)) | |||||
class HomeServer(ReplicationHandler): | |||||
""" A very basic home server implentation that allows people to join a | |||||
room and then invite other people. | |||||
""" | |||||
def __init__(self, server_name, replication_layer, output): | |||||
self.server_name = server_name | |||||
self.replication_layer = replication_layer | |||||
self.replication_layer.set_handler(self) | |||||
self.joined_rooms = {} | |||||
self.output = output | |||||
def on_receive_pdu(self, pdu): | |||||
""" We just received a PDU | |||||
""" | |||||
pdu_type = pdu.pdu_type | |||||
if pdu_type == "sy.room.message": | |||||
self._on_message(pdu) | |||||
elif pdu_type == "sy.room.member" and "membership" in pdu.content: | |||||
if pdu.content["membership"] == "join": | |||||
self._on_join(pdu.context, pdu.state_key) | |||||
elif pdu.content["membership"] == "invite": | |||||
self._on_invite(pdu.origin, pdu.context, pdu.state_key) | |||||
else: | |||||
self.output.print_line("#%s (unrec) %s = %s" % | |||||
(pdu.context, pdu.pdu_type, json.dumps(pdu.content)) | |||||
) | |||||
#def on_state_change(self, pdu): | |||||
##self.output.print_line("#%s (state) %s *** %s" % | |||||
##(pdu.context, pdu.state_key, pdu.pdu_type) | |||||
##) | |||||
#if "joinee" in pdu.content: | |||||
#self._on_join(pdu.context, pdu.content["joinee"]) | |||||
#elif "invitee" in pdu.content: | |||||
#self._on_invite(pdu.origin, pdu.context, pdu.content["invitee"]) | |||||
def _on_message(self, pdu): | |||||
""" We received a message | |||||
""" | |||||
self.output.print_line("#%s %s %s" % | |||||
(pdu.context, pdu.content["sender"], pdu.content["body"]) | |||||
) | |||||
def _on_join(self, context, joinee): | |||||
""" Someone has joined a room, either a remote user or a local user | |||||
""" | |||||
room = self._get_or_create_room(context) | |||||
room.add_participant(joinee) | |||||
self.output.print_line("#%s %s %s" % | |||||
(context, joinee, "*** JOINED") | |||||
) | |||||
def _on_invite(self, origin, context, invitee): | |||||
""" Someone has been invited | |||||
""" | |||||
room = self._get_or_create_room(context) | |||||
room.add_invited(invitee) | |||||
self.output.print_line("#%s %s %s" % | |||||
(context, invitee, "*** INVITED") | |||||
) | |||||
if not room.have_got_metadata and origin is not self.server_name: | |||||
logger.debug("Get room state") | |||||
self.replication_layer.get_state_for_context(origin, context) | |||||
room.have_got_metadata = True | |||||
@defer.inlineCallbacks | |||||
def send_message(self, room_name, sender, body): | |||||
""" Send a message to a room! | |||||
""" | |||||
destinations = yield self.get_servers_for_context(room_name) | |||||
try: | |||||
yield self.replication_layer.send_pdu( | |||||
Pdu.create_new( | |||||
context=room_name, | |||||
pdu_type="sy.room.message", | |||||
content={"sender": sender, "body": body}, | |||||
origin=self.server_name, | |||||
destinations=destinations, | |||||
) | |||||
) | |||||
except Exception as e: | |||||
logger.exception(e) | |||||
@defer.inlineCallbacks | |||||
def join_room(self, room_name, sender, joinee): | |||||
""" Join a room! | |||||
""" | |||||
self._on_join(room_name, joinee) | |||||
destinations = yield self.get_servers_for_context(room_name) | |||||
try: | |||||
pdu = Pdu.create_new( | |||||
context=room_name, | |||||
pdu_type="sy.room.member", | |||||
is_state=True, | |||||
state_key=joinee, | |||||
content={"membership": "join"}, | |||||
origin=self.server_name, | |||||
destinations=destinations, | |||||
) | |||||
yield self.replication_layer.send_pdu(pdu) | |||||
except Exception as e: | |||||
logger.exception(e) | |||||
@defer.inlineCallbacks | |||||
def invite_to_room(self, room_name, sender, invitee): | |||||
""" Invite someone to a room! | |||||
""" | |||||
self._on_invite(self.server_name, room_name, invitee) | |||||
destinations = yield self.get_servers_for_context(room_name) | |||||
try: | |||||
yield self.replication_layer.send_pdu( | |||||
Pdu.create_new( | |||||
context=room_name, | |||||
is_state=True, | |||||
pdu_type="sy.room.member", | |||||
state_key=invitee, | |||||
content={"membership": "invite"}, | |||||
origin=self.server_name, | |||||
destinations=destinations, | |||||
) | |||||
) | |||||
except Exception as e: | |||||
logger.exception(e) | |||||
def paginate(self, room_name, limit=5): | |||||
room = self.joined_rooms.get(room_name) | |||||
if not room: | |||||
return | |||||
dest = room.oldest_server | |||||
return self.replication_layer.paginate(dest, room_name, limit) | |||||
def _get_room_remote_servers(self, room_name): | |||||
return [i for i in self.joined_rooms.setdefault(room_name,).servers] | |||||
def _get_or_create_room(self, room_name): | |||||
return self.joined_rooms.setdefault(room_name, Room(room_name)) | |||||
def get_servers_for_context(self, context): | |||||
return defer.succeed( | |||||
self.joined_rooms.setdefault(context, Room(context)).servers | |||||
) | |||||
def main(stdscr): | |||||
parser = argparse.ArgumentParser() | |||||
parser.add_argument('user', type=str) | |||||
parser.add_argument('-v', '--verbose', action='count') | |||||
args = parser.parse_args() | |||||
user = args.user | |||||
server_name = origin_from_ucid(user) | |||||
## Set up logging ## | |||||
root_logger = logging.getLogger() | |||||
formatter = logging.Formatter('%(asctime)s - %(name)s - %(lineno)d - ' | |||||
'%(levelname)s - %(message)s') | |||||
if not os.path.exists("logs"): | |||||
os.makedirs("logs") | |||||
fh = logging.FileHandler("logs/%s" % user) | |||||
fh.setFormatter(formatter) | |||||
root_logger.addHandler(fh) | |||||
root_logger.setLevel(logging.DEBUG) | |||||
# Hack: The only way to get it to stop logging to sys.stderr :( | |||||
log.theLogPublisher.observers = [] | |||||
observer = log.PythonLoggingObserver() | |||||
observer.start() | |||||
## Set up synapse server | |||||
curses_stdio = cursesio.CursesStdIO(stdscr) | |||||
input_output = InputOutput(curses_stdio, user) | |||||
curses_stdio.set_callback(input_output) | |||||
app_hs = SynapseHomeServer(server_name, db_name="dbs/%s" % user) | |||||
replication = app_hs.get_replication_layer() | |||||
hs = HomeServer(server_name, replication, curses_stdio) | |||||
input_output.set_home_server(hs) | |||||
## Add input_output logger | |||||
io_logger = IOLoggerHandler(input_output) | |||||
io_logger.setFormatter(formatter) | |||||
root_logger.addHandler(io_logger) | |||||
## Start! ## | |||||
try: | |||||
port = int(server_name.split(":")[1]) | |||||
except: | |||||
port = 12345 | |||||
app_hs.get_http_server().start_listening(port) | |||||
reactor.addReader(curses_stdio) | |||||
reactor.run() | |||||
if __name__ == "__main__": | |||||
curses.wrapper(main) |