diff --git a/README.md b/README.md
index 0a4d7d4..471699d 100644
--- a/README.md
+++ b/README.md
@@ -13,17 +13,18 @@ A Pastebin for Tor
## What is this?
-TorPaste is a simple Pastebin service written in Python using the Flask framework.
-It is targeted to users inside Tor and can be easily setup as a Hidden Service.
-As of version v0.4 TorPaste supports multiple backends for storage of data, however
-currently only the local filesystem backend is implemented. TorPaste has been designed
-in order to need no cookies or JavaScript and to run without problems in the Tor Browser
-with the Security and Privacy Settings set to Maximum.
+TorPaste is a simple Pastebin service written in Python using the Flask
+framework. It is targeted to users inside Tor and can be easily setup as a
+Hidden Service. As of version v0.4 TorPaste supports multiple backends for
+storage of data, however currently only the local filesystem and Redis backends
+are implemented. TorPaste has been designed in order to need no cookies or
+JavaScript and to run without problems in the Tor Browser with the Security and
+Privacy Settings set to Maximum.
Unfortunately, the lack of client-side code means all the pastes are stored in
-**plaintext** format, readable by anyone, including the server. For this reason,
-all pastes are indexed and available publicly by default to anyone to see as well.
-Do not use this service for sensitive data.
+**plaintext** format, readable by anyone, including the server. For this
+reason, all pastes are indexed and available publicly by default to anyone to
+see as well. Do not use this service for sensitive data.
## How to run this?
@@ -58,9 +59,10 @@ docker run -d -p 80:80 daknob/torpaste
If you want to run TorPaste in production
( [don't worry, you're not alone](https://paste.daknob.net) ), consider using
a specific tag such as `daknob/torpaste:v0.3`. The `latest` tag is synchronized
-automatically with the `master` branch of this repo and therefore is the bleeding
-edge version. In both cases, don't forget to update your version of TorPaste for
-bug fixes, security patches, and new features. This can be done by running:
+automatically with the `master` branch of this repo and therefore is the
+bleeding edge version. In both cases, don't forget to update your version of
+TorPaste for bug fixes, security patches, and new features. This can be done by
+running:
```bash
docker pull daknob/torpaste
@@ -73,69 +75,88 @@ or of course, for a specific version:
docker pull daknob/torpaste:v0.3
```
-and then stop the previous container and start a new one. It is important to use
-the same settings when launching a new container, so any `-p` / `-e` / `-v` arguments
-need to be specified again.
+and then stop the previous container and start a new one. It is important to
+use the same settings when launching a new container, so any `-p` / `-e` / `-v`
+arguments need to be specified again.
-If you're using Docker and you need the pastes to persist, you can mount the paste
-directory to the local filesystem. This will store all pastes in the host and not
-inside the container. This can be done as such:
+If you're using Docker and you need the pastes to persist, you can mount the
+paste directory to the local filesystem. This will store all pastes in the host
+and not inside the container. This can be done as such:
```bash
docker run -d -p 80:80 -v /path/to/host/:/torpaste/pastes daknob/torpaste
```
## Backends
-TorPaste is extensible and supports multiple backends for storage of its data. As
-of now, the only one implemented is the `filesystem` backend, which stores all data
-as files in the local filesystem. If you're interested in writting a backend, please
-see [Issue #15](https://github.com/DaKnOb/TorPaste/issues/15) for some ideas. For
-more information in the development of new backends, there's an `example.py` file
-inside the `backends` folder which you can copy and start editing right away. The
-file includes a lot of useful information and design documentation for your new
-backend, but if you still want to look at an example, the `backends/filesystem.py`
-is there as well to have a look.
+TorPaste is extensible and supports multiple backends for storage of its data.
+As of now, the only ones implemented are the `filesystem` and `redis` backends.
+The first stores all data as files in the local filesystem while the second
+stores everything in a Redis instance. If you're interested in writting a
+backend, please see [Issue #15](https://github.com/DaKnOb/TorPaste/issues/15)
+for some ideas. For more information in the development of new backends,
+there's an `example.py` file inside the `backends` folder which you can copy
+and start editing right away. The file includes a lot of useful information and
+design documentation for your new backend, but if you still want to look at an
+example, the `backends/filesystem.py` is there as well to have a look.
### filesystem
-This is the first backend available for TorPaste and stores everything in the local
-filesystem. TorPaste versions prior to v0.4 had this backend hardcoded and therefore
-this is an improved implementation so we can maintain backwards compatibility without
-running any migration scripts. It is also the simplest backend and it is used by
-default.
+This is the first backend available for TorPaste and stores everything in the
+local filesystem. TorPaste versions prior to v0.4 had this backend hardcoded
+and therefore this is an improved implementation so we can maintain backwards
+compatibility without running any migration scripts. It is also the simplest
+backend and it is used by default.
+
+### redis
+This is the second backend ever to be added to TorPaste. This backend requires
+a Redis instance to save data to. It uses a single Redis database for all data
+storage and is much faster than the `filesystem` backend according to some
+initial benchmarks. In addition to that, it can be set up in a HA environment
+by using Redis' HA features. Please note that Redis may store data only into
+memory so if you want your data to persist accross reboots / crashes you have
+to enable persistence according to the Redis documentation.
## Configuration
-TorPaste can be configured by using `ENV`ironment Variables. The list of available
-variables as well as their actions is below so you can use them to parameterize your
-installation of the software. Please note that all these variables have a default
-value which may not work well for you, but makes them all optional.
+TorPaste can be configured by using `ENV`ironment Variables. The list of
+available variables as well as their actions is below so you can use them to
+parameterize your installation of the software. Please note that all these
+variables have a default value which may not work well for you, but makes them
+all optional.
### Available ENV Variables
-* `TP_WEBSITE_TITLE` : Use this variable to set the TorPaste Title inside the HTML
-`` tags. *Default:* `Tor Paste`
+* `TP_WEBSITE_TITLE` : Use this variable to set the TorPaste Title inside the
+HTML `` tags. *Default:* `Tor Paste`
* `TP_BACKEND` : Use this variable to select a backend for TorPaste to use. The
-available backends for each version are included in the `COMPATIBLE_BACKENDS` variable
-inside `torpaste.py`. *Default:* `filesystem`
-* `TP_PASTE_MAX_SIZE` : Use this variable to set the maximum paste size, in bytes. The
-possible values are formatted as ` `, for example `10 M`, or `128 B`,
-or `16 k`. Any value that starts with `0` changes this limit to unlimited. *Default:*
-`0`
-* `TP_PASTE_LIST_ACTIVE` : Use this variable to enable or disable the paste listing
-available in the `Pastes` menu. *Default:* `True`
-* `TP_CSP_REPORT_URI` : Use this variable to set a `report-uri` for the Content Security
-Policy of TorPaste. If this variable is not set, no `report-uri` is added, which is the
-default behavior.
-* `TP_ENABLED_PASTE_VISIBILITIES` : Use this variable to select the available paste
-visibilities, separated by a comma. Example: "public,unlisted". The available backends
-for each version are included in the `AVAILABLE_VISIBILITIES` variable inside
-`torpaste.py`. *Default:* `public`.
+available backends for each version are included in the `COMPATIBLE_BACKENDS`
+variable inside `torpaste.py`. *Default:* `filesystem`
+* `TP_PASTE_MAX_SIZE` : Use this variable to set the maximum paste size, in
+bytes. The possible values are formatted as ` `, for example
+`10 M`, or `128 B`, or `16 k`. Any value that starts with `0` changes this
+limit to unlimited. *Default:* `0`
+* `TP_PASTE_LIST_ACTIVE` : Use this variable to enable or disable the paste
+listing available in the `Pastes` menu. *Default:* `True`
+* `TP_CSP_REPORT_URI` : Use this variable to set a `report-uri` for the Content
+Security Policy of TorPaste. If this variable is not set, no `report-uri` is
+added, which is the default behavior.
+* `TP_ENABLED_PASTE_VISIBILITIES` : Use this variable to select the available
+paste visibilities, separated by a comma. Example: "public,unlisted". The
+available backends for each version are included in the
+`AVAILABLE_VISIBILITIES` variable inside `torpaste.py`. *Default:* `public`.
### Backend ENV Variables
-Each backend may need one or more additional `ENV` variables to work. For example,
-a MySQL backend may need the `HOST`, `PORT`, `USERNAME`, and `PASSWORD` to connect
-to the database. To prevent conflicts, all these variables will be available as
-`TP_BACKEND_BACKENDNAME_VARIABLE` where `BACKENDNAME` is the name of the backend,
-such as `MYSQL` and the `VARIABLE` will be the name of the variable, such as `HOST`.
-
-Currently there are no used backend `ENV` variables. When there are, you will find
-a list of all backends and their variables here.
+Each backend may need one or more additional `ENV` variables to work. For
+example, a MySQL backend may need the `HOST`, `PORT`, `USERNAME`, and
+`PASSWORD` to connect to the database. To prevent conflicts, all these
+variables will be available as `TP_BACKEND_BACKENDNAME_VARIABLE` where
+`BACKENDNAME` is the name of the backend, such as `MYSQL` and the `VARIABLE`
+will be the name of the variable, such as `HOST`.
+
+#### redis
+* `TP_BACKEND_REDIS_HOST` : The Redis server running to connect to. *Default:*
+`127.0.0.1`
+* `TP_BACKEND_REDIS_PORT` : The Redis server port to connect to. *Default:*
+`6379`
+* `TP_BACKEND_REDIS_PASSWORD` : The Redis server password needed for
+authentication. *Default: none*
+* `TP_BACKEND_REDIS_DB_INDEX` : The database index to connect to such as `0` or
+`1`. *Default:* `1`.
diff --git a/backends/redis.py b/backends/redis.py
new file mode 100644
index 0000000..4f32a54
--- /dev/null
+++ b/backends/redis.py
@@ -0,0 +1,183 @@
+#!../bin/python
+
+import backends.exceptions as e
+import redis
+from os import getenv
+
+
+def initialize_backend():
+ """
+ This method is called when the Flask application starts.
+ Here you can do any initialization that may be required,
+ such as connecting to a remote database or making sure a file exists.
+ """
+ global rb
+
+ RedisHost = getenv("TP_BACKEND_REDIS_HOST") or "127.0.0.1"
+ RedisPort = int(getenv("TP_BACKEND_REDIS_PORT") or "6379")
+ RedisPass = getenv("TP_BACKEND_REDIS_PASSWORD") or None
+ RedisDBI = int(getenv("TP_BACKEND_REDIS_DB_INDEX") or "1")
+
+ rb = redis.StrictRedis(
+ host=RedisHost,
+ port=RedisPort,
+ password=RedisPass,
+ db=RedisDBI
+ )
+
+ if rb.ping() is not True:
+ raise e.ErrorException(
+ "Could not connect to the backend database. Please try again" +
+ "later. If the error persists, please contact a system" +
+ "administrator."
+ )
+
+ rb.save()
+
+ return
+
+
+def new_paste(paste_id, paste_content):
+ """
+ This method is called when the Flask application wants to create a new
+ paste. The arguments given are the Paste ID, which is a paste identifier
+ that is not necessarily unique, as well as the paste content. Please note
+ that the paste must be retrievable given the above Paste ID as well as
+ he fact that paste_id is (typically) an ASCII string while paste_content
+ can (and will) contain UTF-8 characters.
+ :param paste_id: a not necessarily unique id of the paste
+ :param paste_content: content of the paste (utf-8 encoded)
+ :return:
+ """
+
+ rb.set(paste_id, paste_content)
+
+ rb.save()
+
+ return
+
+
+def update_paste_metadata(paste_id, metadata):
+ """
+ This method is called by the Flask application to update a paste's
+ metadata. For this to happen, the application passes the Paste ID,
+ which is typically an ASCII string, as well as a Python dictionary
+ that contains the new metadata. This method must overwrite any and
+ all metadata with the passed dictionary. For example, if a paste has
+ the keys a and b and this method is called with only keys b and c,
+ the final metadata must be b and c only, and not a.
+ :param paste_id: ASCII coded id of the paste
+ :param metadata: dictionary containing the metadata
+ :return:
+ """
+
+ for md in rb.keys(paste_id + ".*"):
+ rb.delete(md)
+ for md in metadata:
+ rb.set(paste_id + "." + md, metadata[md])
+
+ rb.save()
+
+ return
+
+
+def does_paste_exist(paste_id):
+ """
+ This method is called when the Flask application wants to check if a
+ paste with a given Paste ID exists. The Paste ID is (typically) an
+ ASCII string and your method must return True if a paste with this ID
+ exists, or False if it doesn't.
+ :param paste_id: ASCII string which represents the ID of the paste
+ :return: True if paste with given ID exists, false otherwise
+ """
+
+ return rb.exists(paste_id)
+
+
+def get_paste_contents(paste_id):
+ """
+ This method must return all the paste contents in UTF-8 encoding for
+ a given Paste ID. The Paste ID is typically in ASCII, and it is
+ guaranteed that this Paste ID exists.
+ :param paste_id: ASCII string which represents the ID of the paste
+ :return: the content of the paste in UTF-8 encoding
+ """
+
+ return rb.get(paste_id).decode("utf-8")
+
+
+def get_paste_metadata(paste_id):
+ """
+ This method must return a Python Dictionary with all the currently
+ stored metadata for the paste with the given Paste ID. All keys of
+ the dictionary are typically in ASCII, while all values are in
+ UTF-8. It is guaranteed that the Paste ID exists.
+ :param paste_id: ASCII string which represents the ID of the paste
+ :return: a dictionary with the metadata of a given paste
+ """
+
+ ret = {}
+
+ f = rb.keys(paste_id + ".*")
+ for md in f:
+ ret[md.split(".")[1]] = rb.get(md).decode("utf-8")
+
+ return str(ret)
+
+
+def get_paste_metadata_value(paste_id, key):
+ """
+ This method must return the value of the metadata key provided for
+ the paste whose Paste ID is provided. If the key is not set, the
+ method should return None. You can assume that a paste with this
+ Paste ID exists, and you can also assume that both parameters
+ passed are typically ASCII.
+ :param paste_id: ASCII string which represents the ID of the paste
+ :param key: key of the metadata
+ :return: value of the metadata key provided for the given ID, None if
+ the key wasn't set
+ """
+
+ if rb.exists(paste_id + "." + key) is not True:
+ return None
+ else:
+ return rb.get(paste_id + "." + key).decode("utf-8")
+
+
+def get_all_paste_ids(filters={}, fdefaults={}):
+ """
+ This method must return a Python list containing the ASCII ID of all
+ pastes which match the (optional) filters provided. The order does not
+ matter so it can be the same or it can be different every time this
+ function is called. In the case of no pastes, the method must return a
+ Python list with a single item, whose content must be equal to 'none'.
+ :param filters: a dictionary of filters
+ :param fdefaults: a dictionary with the default value for each filter
+ if it's not present
+ :return: a list containing all paste IDs
+ """
+
+ ak = rb.keys()
+ ap = []
+ for k in ak:
+ if b"." not in k:
+ ap.append(k.decode("utf-8"))
+ filt = []
+ for p in ap:
+ keep = True
+
+ for k, v in filters.items():
+ gpmv = get_paste_metadata_value(p, k)
+ if gpmv is None:
+ gpmv = fdefaults[k]
+ if gpmv != v:
+ keep = False
+ break
+
+ if keep:
+ filt.append(p)
+
+ if len(filt) == 0:
+ filt = ['none']
+
+ return filt
diff --git a/requirements.txt b/requirements.txt
index e3e9a71..1fb0bd9 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1 +1,2 @@
Flask
+redis
diff --git a/torpaste.py b/torpaste.py
index 3b54153..ef40e37 100755
--- a/torpaste.py
+++ b/torpaste.py
@@ -17,7 +17,7 @@
VERSION = check_output(["git", "describe"]).decode("utf-8").replace("\n", "")
# Compatible Backends List
-COMPATIBLE_BACKENDS = ["filesystem"]
+COMPATIBLE_BACKENDS = ["filesystem", "redis"]
# Available list of paste visibilities
# public: can be viewed by all, is listed in /list