@@ -1,14 +1,96 @@ | |||
About | |||
===== | |||
Matrix is an ambitious new ecosystem for open federated Instant Messaging and VoIP[1]. | |||
Matrix specifies a set of pragmatic RESTful HTTP JSON APIs as an open standard, providing: | |||
- Creating and managing fully distributed chat rooms with no | |||
single points of control or failure | |||
- Eventually-consistent cryptographically secure synchronisation of room | |||
state across a global open network of federated servers and services | |||
- Sending and receiving extensible messages in a room with (optional) | |||
end-to-end encryption[2] | |||
- Inviting, joining, leaving, kicking, banning room members | |||
- Managing user accounts (registration, login, logout) | |||
- Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers, | |||
Facebook accounts to authenticate, identify and discover users on Matrix. | |||
- Placing 1:1 VoIP and Video calls (in development) | |||
These APIs are intended to be implemented on a wide range of servers, services | |||
and clients which then form the Matrix ecosystem, and allow developers to build | |||
messaging and VoIP functionality on top of the open Matrix community rather than | |||
using closed or proprietary solutions. The hope is for Matrix to act as the | |||
building blocks for a new generation of fully open and interoperable messaging | |||
and VoIP apps for the internet. | |||
Synapse is a reference "homeserver" implementation of Matrix from the core | |||
development team at matrix.org, written in Python/Twisted for clarity and | |||
simplicity. It is intended to showcase the concept of Matrix and let folks see | |||
the spec in the context of a codebase and let you run your own homeserver and | |||
generally help bootstrap the ecosystem. | |||
In Matrix, every user runs one or more Matrix clients, which connect through to | |||
a Matrix homeserver which stores all their personal chat history and user | |||
account information - much as a mail client connects through to an IMAP/SMTP | |||
server. Just like email, you can either run your own Matrix homeserver and | |||
control and own your own communications and history or use one hosted by someone | |||
else (e.g. matrix.org) - there is no single point of control or mandatory | |||
service provider in Matrix, unlike WhatsApp, Facebook, Hangouts, etc. | |||
Synapse ships with two basic demo Matrix clients: webclient (a basic group chat web client demo implemented in AngularJS) and cmdclient (a basic Python commandline utility which lets you easily see what the JSON APIs are up to). | |||
We'd like to invite you to take a look at the Matrix spec, try to run a homeserver, and join the existing Matrix chatrooms already out there, experiment with the APIs and the demo clients, and let us know your thoughts at https://github.com/matrix-org/synapse/issues or at matrix@matrix.org. | |||
Thanks for trying Matrix! | |||
[1] VoIP currently in development | |||
[2] End-to-end encryption is currently in development | |||
Directory Structure | |||
=================== | |||
:: | |||
. | |||
├── cmdclient Basic CLI python Matrix client | |||
├── demo Scripts for running standalone Matrix demos | |||
├── docs All doc, including the draft Matrix API spec | |||
│ ├── client-server The client-server Matrix API spec | |||
│ ├── model Domain-specific elements of the Matrix API spec | |||
│ ├── server-server The server-server model of the Matrix API spec | |||
│ └── sphinx The internal API doc of the Synapse homeserver | |||
├── experiments Early experiments of using Synapse's internal APIs | |||
├── graph Visualisation of Matrix's distributed message store | |||
├── synapse The reference Matrix homeserver implementation | |||
│ ├── api Common building blocks for the APIs | |||
│ │ ├── events Definition of state representation Events | |||
│ │ └── streams Definition of streamable Event objects | |||
│ ├── app The __main__ entry point for the homeserver | |||
│ ├── crypto The PKI client/server used for secure federation | |||
│ │ └── resource PKI helper objects (e.g. keys) | |||
│ ├── federation Server-server state replication logic | |||
│ ├── handlers The main business logic of the homeserver | |||
│ ├── http Wrappers around Twisted's HTTP server & client | |||
│ ├── rest Servlet-style RESTful API | |||
│ ├── storage Persistence subsystem (currently only sqlite3) | |||
│ │ └── schema sqlite persistence schema | |||
│ └── util Synapse-specific utilities | |||
├── tests Unit tests for the Synapse homeserver | |||
└── webclient Basic AngularJS Matrix web client | |||
Installation | |||
============ | |||
[TODO(kegan): I also needed libffi-dev, which I don't think is included in build-essential.] | |||
First, the dependencies need to be installed. Start by installing | |||
'python2.7-dev' and the various tools of the compiler toolchain. | |||
First, the dependencies need to be installed. Start by installing 'python-dev' | |||
and the various tools of the compiler toolchain: | |||
N.B. that python 2.x where x >= 7 is required. | |||
Installing prerequisites on ubuntu:: | |||
$ sudo apt-get install build-essential python-dev | |||
$ sudo apt-get install build-essential python2.7-dev libffi-dev | |||
Installing prerequisites on Mac OS X:: | |||
@@ -35,12 +117,13 @@ This should end with a 'PASSED' result:: | |||
PASSED (successes=143) | |||
Running The Home Server | |||
======================= | |||
Running The Synapse Homeserver | |||
============================== | |||
In order for other home servers to send messages to your server, they will need | |||
to know its host name. You have two choices here, which will influence the form | |||
of your user IDs: | |||
In order for other homeservers to send messages to your server, it will need to | |||
be publicly visible on the internet, and they will need to know its host name. | |||
You have two choices here, which will influence the form of your matrix user | |||
IDs: | |||
1) Use the machine's own hostname as available on public DNS in the form of its | |||
A or AAAA records. This is easier to set up initially, perhaps for testing, | |||
@@ -58,10 +141,9 @@ For the first form, simply pass the required hostname (of the machine) as the | |||
For the second form, first create your SRV record and publish it in DNS. This | |||
needs to be named _matrix._tcp.YOURDOMAIN, and point at at least one hostname | |||
and port where the server is running. (At the current time we only support a | |||
single server, but we may at some future point support multiple servers, for | |||
backup failover or load-balancing purposes). The DNS record would then look | |||
something like:: | |||
and port where the server is running. (At the current time synapse does not | |||
support clustering multiple servers into a single logical homeserver). The DNS | |||
record would then look something like:: | |||
_matrix._tcp IN SRV 10 0 8448 machine.my.domain.name. | |||
@@ -73,9 +155,13 @@ SRV record, as that is the name other machines will expect it to have:: | |||
You may additionally want to pass one or more "-v" options, in order to | |||
increase the verbosity of logging output; at least for initial testing. | |||
For the initial alpha release, the homeserver is not speaking TLS for | |||
either client-server or server-server traffic for ease of debugging. We have | |||
also not spent any time yet getting the homeserver to run behind loadbalancers. | |||
Running The Web Client | |||
====================== | |||
Running The Demo Web Client | |||
=========================== | |||
At the present time, the web client is not directly served by the homeserver's | |||
HTTP server. To serve this in a form the web browser can reach, arrange for the | |||
@@ -92,6 +178,7 @@ HTML5 local storage to remember its config), you will need to log in to your | |||
account. If you don't yet have an account, because you've just started the | |||
homeserver for the first time, then you'll need to register one. | |||
Registering A New Account | |||
------------------------- | |||
@@ -103,7 +190,10 @@ account. Your name will take the form of:: | |||
(pronounced "at localpart on my dot domain dot here") | |||
Specify your desired localpart in the topmost box of the "Register for an | |||
account" form, and click the "Register" button. | |||
account" form, and click the "Register" button. Hostnames can contain ports if | |||
required due to lack of SRV records (e.g. @matthew:localhost:8080 on an internal | |||
synapse sandbox running on localhost) | |||
Logging In To An Existing Account | |||
--------------------------------- | |||
@@ -0,0 +1,154 @@ | |||
import curses | |||
import curses.wrapper | |||
from curses.ascii import isprint | |||
from twisted.internet import reactor | |||
class CursesStdIO(): | |||
def __init__(self, stdscr, callback=None): | |||
self.statusText = "Synapse test app -" | |||
self.searchText = '' | |||
self.stdscr = stdscr | |||
self.logLine = '' | |||
self.callback = callback | |||
self._setup() | |||
def _setup(self): | |||
self.stdscr.nodelay(1) # Make non blocking | |||
self.rows, self.cols = self.stdscr.getmaxyx() | |||
self.lines = [] | |||
curses.use_default_colors() | |||
self.paintStatus(self.statusText) | |||
self.stdscr.refresh() | |||
def set_callback(self, callback): | |||
self.callback = callback | |||
def fileno(self): | |||
""" We want to select on FD 0 """ | |||
return 0 | |||
def connectionLost(self, reason): | |||
self.close() | |||
def print_line(self, text): | |||
""" add a line to the internal list of lines""" | |||
self.lines.append(text) | |||
self.redraw() | |||
def print_log(self, text): | |||
self.logLine = text | |||
self.redraw() | |||
def redraw(self): | |||
""" method for redisplaying lines | |||
based on internal list of lines """ | |||
self.stdscr.clear() | |||
self.paintStatus(self.statusText) | |||
i = 0 | |||
index = len(self.lines) - 1 | |||
while i < (self.rows - 3) and index >= 0: | |||
self.stdscr.addstr(self.rows - 3 - i, 0, self.lines[index], | |||
curses.A_NORMAL) | |||
i = i + 1 | |||
index = index - 1 | |||
self.printLogLine(self.logLine) | |||
self.stdscr.refresh() | |||
def paintStatus(self, text): | |||
if len(text) > self.cols: | |||
raise RuntimeError("TextTooLongError") | |||
self.stdscr.addstr( | |||
self.rows - 2, 0, | |||
text + ' ' * (self.cols - len(text)), | |||
curses.A_STANDOUT) | |||
def printLogLine(self, text): | |||
self.stdscr.addstr( | |||
0, 0, | |||
text + ' ' * (self.cols - len(text)), | |||
curses.A_STANDOUT) | |||
def doRead(self): | |||
""" Input is ready! """ | |||
curses.noecho() | |||
c = self.stdscr.getch() # read a character | |||
if c == curses.KEY_BACKSPACE: | |||
self.searchText = self.searchText[:-1] | |||
elif c == curses.KEY_ENTER or c == 10: | |||
text = self.searchText | |||
self.searchText = '' | |||
self.print_line(">> %s" % text) | |||
try: | |||
if self.callback: | |||
self.callback.on_line(text) | |||
except Exception as e: | |||
self.print_line(str(e)) | |||
self.stdscr.refresh() | |||
elif isprint(c): | |||
if len(self.searchText) == self.cols - 2: | |||
return | |||
self.searchText = self.searchText + chr(c) | |||
self.stdscr.addstr(self.rows - 1, 0, | |||
self.searchText + (' ' * ( | |||
self.cols - len(self.searchText) - 2))) | |||
self.paintStatus(self.statusText + ' %d' % len(self.searchText)) | |||
self.stdscr.move(self.rows - 1, len(self.searchText)) | |||
self.stdscr.refresh() | |||
def logPrefix(self): | |||
return "CursesStdIO" | |||
def close(self): | |||
""" clean up """ | |||
curses.nocbreak() | |||
self.stdscr.keypad(0) | |||
curses.echo() | |||
curses.endwin() | |||
class Callback(object): | |||
def __init__(self, stdio): | |||
self.stdio = stdio | |||
def on_line(self, text): | |||
self.stdio.print_line(text) | |||
def main(stdscr): | |||
screen = CursesStdIO(stdscr) # create Screen object | |||
callback = Callback(screen) | |||
screen.set_callback(callback) | |||
stdscr.refresh() | |||
reactor.addReader(screen) | |||
reactor.run() | |||
screen.close() | |||
if __name__ == '__main__': | |||
curses.wrapper(main) |
@@ -0,0 +1,380 @@ | |||
# -*- coding: utf-8 -*- | |||
""" This is an example of using the server to server implementation to do a | |||
basic chat style thing. It accepts commands from stdin and outputs to stdout. | |||
It assumes that ucids are of the form <user>@<domain>, and uses <domain> as | |||
the address of the remote home server to hit. | |||
Usage: | |||
python test_messaging.py <port> | |||
Currently assumes the local address is localhost:<port> | |||
""" | |||
from synapse.federation import ( | |||
ReplicationHandler | |||
) | |||
from synapse.federation.units import Pdu | |||
from synapse.util import origin_from_ucid | |||
from synapse.app.homeserver import SynapseHomeServer | |||
#from synapse.util.logutils import log_function | |||
from twisted.internet import reactor, defer | |||
from twisted.python import log | |||
import argparse | |||
import json | |||
import logging | |||
import os | |||
import re | |||
import cursesio | |||
import curses.wrapper | |||
logger = logging.getLogger("example") | |||
def excpetion_errback(failure): | |||
logging.exception(failure) | |||
class InputOutput(object): | |||
""" This is responsible for basic I/O so that a user can interact with | |||
the example app. | |||
""" | |||
def __init__(self, screen, user): | |||
self.screen = screen | |||
self.user = user | |||
def set_home_server(self, server): | |||
self.server = server | |||
def on_line(self, line): | |||
""" This is where we process commands. | |||
""" | |||
try: | |||
m = re.match("^join (\S+)$", line) | |||
if m: | |||
# The `sender` wants to join a room. | |||
room_name, = m.groups() | |||
self.print_line("%s joining %s" % (self.user, room_name)) | |||
self.server.join_room(room_name, self.user, self.user) | |||
#self.print_line("OK.") | |||
return | |||
m = re.match("^invite (\S+) (\S+)$", line) | |||
if m: | |||
# `sender` wants to invite someone to a room | |||
room_name, invitee = m.groups() | |||
self.print_line("%s invited to %s" % (invitee, room_name)) | |||
self.server.invite_to_room(room_name, self.user, invitee) | |||
#self.print_line("OK.") | |||
return | |||
m = re.match("^send (\S+) (.*)$", line) | |||
if m: | |||
# `sender` wants to message a room | |||
room_name, body = m.groups() | |||
self.print_line("%s send to %s" % (self.user, room_name)) | |||
self.server.send_message(room_name, self.user, body) | |||
#self.print_line("OK.") | |||
return | |||
m = re.match("^paginate (\S+)$", line) | |||
if m: | |||
# we want to paginate a room | |||
room_name, = m.groups() | |||
self.print_line("paginate %s" % room_name) | |||
self.server.paginate(room_name) | |||
return | |||
self.print_line("Unrecognized command") | |||
except Exception as e: | |||
logger.exception(e) | |||
def print_line(self, text): | |||
self.screen.print_line(text) | |||
def print_log(self, text): | |||
self.screen.print_log(text) | |||
class IOLoggerHandler(logging.Handler): | |||
def __init__(self, io): | |||
logging.Handler.__init__(self) | |||
self.io = io | |||
def emit(self, record): | |||
if record.levelno < logging.WARN: | |||
return | |||
msg = self.format(record) | |||
self.io.print_log(msg) | |||
class Room(object): | |||
""" Used to store (in memory) the current membership state of a room, and | |||
which home servers we should send PDUs associated with the room to. | |||
""" | |||
def __init__(self, room_name): | |||
self.room_name = room_name | |||
self.invited = set() | |||
self.participants = set() | |||
self.servers = set() | |||
self.oldest_server = None | |||
self.have_got_metadata = False | |||
def add_participant(self, participant): | |||
""" Someone has joined the room | |||
""" | |||
self.participants.add(participant) | |||
self.invited.discard(participant) | |||
server = origin_from_ucid(participant) | |||
self.servers.add(server) | |||
if not self.oldest_server: | |||
self.oldest_server = server | |||
def add_invited(self, invitee): | |||
""" Someone has been invited to the room | |||
""" | |||
self.invited.add(invitee) | |||
self.servers.add(origin_from_ucid(invitee)) | |||
class HomeServer(ReplicationHandler): | |||
""" A very basic home server implentation that allows people to join a | |||
room and then invite other people. | |||
""" | |||
def __init__(self, server_name, replication_layer, output): | |||
self.server_name = server_name | |||
self.replication_layer = replication_layer | |||
self.replication_layer.set_handler(self) | |||
self.joined_rooms = {} | |||
self.output = output | |||
def on_receive_pdu(self, pdu): | |||
""" We just received a PDU | |||
""" | |||
pdu_type = pdu.pdu_type | |||
if pdu_type == "sy.room.message": | |||
self._on_message(pdu) | |||
elif pdu_type == "sy.room.member" and "membership" in pdu.content: | |||
if pdu.content["membership"] == "join": | |||
self._on_join(pdu.context, pdu.state_key) | |||
elif pdu.content["membership"] == "invite": | |||
self._on_invite(pdu.origin, pdu.context, pdu.state_key) | |||
else: | |||
self.output.print_line("#%s (unrec) %s = %s" % | |||
(pdu.context, pdu.pdu_type, json.dumps(pdu.content)) | |||
) | |||
#def on_state_change(self, pdu): | |||
##self.output.print_line("#%s (state) %s *** %s" % | |||
##(pdu.context, pdu.state_key, pdu.pdu_type) | |||
##) | |||
#if "joinee" in pdu.content: | |||
#self._on_join(pdu.context, pdu.content["joinee"]) | |||
#elif "invitee" in pdu.content: | |||
#self._on_invite(pdu.origin, pdu.context, pdu.content["invitee"]) | |||
def _on_message(self, pdu): | |||
""" We received a message | |||
""" | |||
self.output.print_line("#%s %s %s" % | |||
(pdu.context, pdu.content["sender"], pdu.content["body"]) | |||
) | |||
def _on_join(self, context, joinee): | |||
""" Someone has joined a room, either a remote user or a local user | |||
""" | |||
room = self._get_or_create_room(context) | |||
room.add_participant(joinee) | |||
self.output.print_line("#%s %s %s" % | |||
(context, joinee, "*** JOINED") | |||
) | |||
def _on_invite(self, origin, context, invitee): | |||
""" Someone has been invited | |||
""" | |||
room = self._get_or_create_room(context) | |||
room.add_invited(invitee) | |||
self.output.print_line("#%s %s %s" % | |||
(context, invitee, "*** INVITED") | |||
) | |||
if not room.have_got_metadata and origin is not self.server_name: | |||
logger.debug("Get room state") | |||
self.replication_layer.get_state_for_context(origin, context) | |||
room.have_got_metadata = True | |||
@defer.inlineCallbacks | |||
def send_message(self, room_name, sender, body): | |||
""" Send a message to a room! | |||
""" | |||
destinations = yield self.get_servers_for_context(room_name) | |||
try: | |||
yield self.replication_layer.send_pdu( | |||
Pdu.create_new( | |||
context=room_name, | |||
pdu_type="sy.room.message", | |||
content={"sender": sender, "body": body}, | |||
origin=self.server_name, | |||
destinations=destinations, | |||
) | |||
) | |||
except Exception as e: | |||
logger.exception(e) | |||
@defer.inlineCallbacks | |||
def join_room(self, room_name, sender, joinee): | |||
""" Join a room! | |||
""" | |||
self._on_join(room_name, joinee) | |||
destinations = yield self.get_servers_for_context(room_name) | |||
try: | |||
pdu = Pdu.create_new( | |||
context=room_name, | |||
pdu_type="sy.room.member", | |||
is_state=True, | |||
state_key=joinee, | |||
content={"membership": "join"}, | |||
origin=self.server_name, | |||
destinations=destinations, | |||
) | |||
yield self.replication_layer.send_pdu(pdu) | |||
except Exception as e: | |||
logger.exception(e) | |||
@defer.inlineCallbacks | |||
def invite_to_room(self, room_name, sender, invitee): | |||
""" Invite someone to a room! | |||
""" | |||
self._on_invite(self.server_name, room_name, invitee) | |||
destinations = yield self.get_servers_for_context(room_name) | |||
try: | |||
yield self.replication_layer.send_pdu( | |||
Pdu.create_new( | |||
context=room_name, | |||
is_state=True, | |||
pdu_type="sy.room.member", | |||
state_key=invitee, | |||
content={"membership": "invite"}, | |||
origin=self.server_name, | |||
destinations=destinations, | |||
) | |||
) | |||
except Exception as e: | |||
logger.exception(e) | |||
def paginate(self, room_name, limit=5): | |||
room = self.joined_rooms.get(room_name) | |||
if not room: | |||
return | |||
dest = room.oldest_server | |||
return self.replication_layer.paginate(dest, room_name, limit) | |||
def _get_room_remote_servers(self, room_name): | |||
return [i for i in self.joined_rooms.setdefault(room_name,).servers] | |||
def _get_or_create_room(self, room_name): | |||
return self.joined_rooms.setdefault(room_name, Room(room_name)) | |||
def get_servers_for_context(self, context): | |||
return defer.succeed( | |||
self.joined_rooms.setdefault(context, Room(context)).servers | |||
) | |||
def main(stdscr): | |||
parser = argparse.ArgumentParser() | |||
parser.add_argument('user', type=str) | |||
parser.add_argument('-v', '--verbose', action='count') | |||
args = parser.parse_args() | |||
user = args.user | |||
server_name = origin_from_ucid(user) | |||
## Set up logging ## | |||
root_logger = logging.getLogger() | |||
formatter = logging.Formatter('%(asctime)s - %(name)s - %(lineno)d - ' | |||
'%(levelname)s - %(message)s') | |||
if not os.path.exists("logs"): | |||
os.makedirs("logs") | |||
fh = logging.FileHandler("logs/%s" % user) | |||
fh.setFormatter(formatter) | |||
root_logger.addHandler(fh) | |||
root_logger.setLevel(logging.DEBUG) | |||
# Hack: The only way to get it to stop logging to sys.stderr :( | |||
log.theLogPublisher.observers = [] | |||
observer = log.PythonLoggingObserver() | |||
observer.start() | |||
## Set up synapse server | |||
curses_stdio = cursesio.CursesStdIO(stdscr) | |||
input_output = InputOutput(curses_stdio, user) | |||
curses_stdio.set_callback(input_output) | |||
app_hs = SynapseHomeServer(server_name, db_name="dbs/%s" % user) | |||
replication = app_hs.get_replication_layer() | |||
hs = HomeServer(server_name, replication, curses_stdio) | |||
input_output.set_home_server(hs) | |||
## Add input_output logger | |||
io_logger = IOLoggerHandler(input_output) | |||
io_logger.setFormatter(formatter) | |||
root_logger.addHandler(io_logger) | |||
## Start! ## | |||
try: | |||
port = int(server_name.split(":")[1]) | |||
except: | |||
port = 12345 | |||
app_hs.get_http_server().start_listening(port) | |||
reactor.addReader(curses_stdio) | |||
reactor.run() | |||
if __name__ == "__main__": | |||
curses.wrapper(main) |