[WIP] Add Redis Backend Support

README.md

@@ -13,17 +13,18 @@ A Pastebin for Tor
 
 ## What is this?
 
-TorPaste is a simple Pastebin service written in Python using the Flask framework.
-It is targeted to users inside Tor and can be easily setup as a Hidden Service.
-As of version v0.4 TorPaste supports multiple backends for storage of data, however
-currently only the local filesystem backend is implemented. TorPaste has been designed
-in order to need no cookies or JavaScript and to run without problems in the Tor Browser
-with the Security and Privacy Settings set to Maximum.
+TorPaste is a simple Pastebin service written in Python using the Flask
+framework. It is targeted to users inside Tor and can be easily setup as a
+Hidden Service. As of version v0.4 TorPaste supports multiple backends for
+storage of data, however currently only the local filesystem and Redis backends
+are implemented. TorPaste has been designed in order to need no cookies or
+JavaScript and to run without problems in the Tor Browser with the Security and
+Privacy Settings set to Maximum.
 
 Unfortunately, the lack of client-side code means all the pastes are stored in
-**plaintext** format, readable by anyone, including the server. For this reason,
-all pastes are indexed and available publicly by default to anyone to see as well.
-Do not use this service for sensitive data.
+**plaintext** format, readable by anyone, including the server. For this
+reason, all pastes are indexed and available publicly by default to anyone to
+see as well. Do not use this service for sensitive data.
 
 ## How to run this?
 
@@ -58,9 +59,10 @@ docker run -d -p 80:80 daknob/torpaste
 If you want to run TorPaste in production
 ( [don't worry, you're not alone](https://paste.daknob.net) ), consider using
 a specific tag such as `daknob/torpaste:v0.3`. The `latest` tag is synchronized
-automatically with the `master` branch of this repo and therefore is the bleeding
-edge version. In both cases, don't forget to update your version of TorPaste for
-bug fixes, security patches, and new features. This can be done by running:
+automatically with the `master` branch of this repo and therefore is the
+bleeding edge version. In both cases, don't forget to update your version of
+TorPaste for bug fixes, security patches, and new features. This can be done by
+running:
 
 ```bash
 docker pull daknob/torpaste
@@ -73,69 +75,88 @@ or of course, for a specific version:
 docker pull daknob/torpaste:v0.3
 ```
 
-and then stop the previous container and start a new one. It is important to use
-the same settings when launching a new container, so any `-p` / `-e` / `-v` arguments
-need to be specified again.
+and then stop the previous container and start a new one. It is important to
+use the same settings when launching a new container, so any `-p` / `-e` / `-v`
+arguments need to be specified again.
 
-If you're using Docker and you need the pastes to persist, you can mount the paste
-directory to the local filesystem. This will store all pastes in the host and not
-inside the container. This can be done as such:
+If you're using Docker and you need the pastes to persist, you can mount the
+paste directory to the local filesystem. This will store all pastes in the host
+and not inside the container. This can be done as such:
 
 ```bash
 docker run -d -p 80:80 -v /path/to/host/:/torpaste/pastes daknob/torpaste
 ```
 
 ## Backends
-TorPaste is extensible and supports multiple backends for storage of its data. As
-of now, the only one implemented is the `filesystem` backend, which stores all data
-as files in the local filesystem. If you're interested in writting a backend, please
-see [Issue #15](https://github.com/DaKnOb/TorPaste/issues/15) for some ideas. For
-more information in the development of new backends, there's an `example.py` file
-inside the `backends` folder which you can copy and start editing right away. The
-file includes a lot of useful information and design documentation for your new
-backend, but if you still want to look at an example, the `backends/filesystem.py`
-is there as well to have a look.
+TorPaste is extensible and supports multiple backends for storage of its data.
+As of now, the only ones implemented are the `filesystem` and `redis` backends.
+The first stores all data as files in the local filesystem while the second
+stores everything in a Redis instance. If you're interested in writting a
+backend, please see [Issue #15](https://github.com/DaKnOb/TorPaste/issues/15)
+for some ideas. For more information in the development of new backends,
+there's an `example.py` file inside the `backends` folder which you can copy
+and start editing right away. The file includes a lot of useful information and
+design documentation for your new backend, but if you still want to look at an
+example, the `backends/filesystem.py` is there as well to have a look.
 
 ### filesystem
-This is the first backend available for TorPaste and stores everything in the local
-filesystem. TorPaste versions prior to v0.4 had this backend hardcoded and therefore
-this is an improved implementation so we can maintain backwards compatibility without
-running any migration scripts. It is also the simplest backend and it is used by
-default.
+This is the first backend available for TorPaste and stores everything in the
+local filesystem. TorPaste versions prior to v0.4 had this backend hardcoded
+and therefore this is an improved implementation so we can maintain backwards
+compatibility without running any migration scripts. It is also the simplest
+backend and it is used by default.
+
+### redis
+This is the second backend ever to be added to TorPaste. This backend requires
+a Redis instance to save data to. It uses a single Redis database for all data
+storage and is much faster than the `filesystem` backend according to some
+initial benchmarks. In addition to that, it can be set up in a HA environment
+by using Redis' HA features. Please note that Redis may store data only into
+memory so if you want your data to persist accross reboots / crashes you have
+to enable persistence according to the Redis documentation.
 
 ## Configuration
-TorPaste can be configured by using `ENV`ironment Variables. The list of available
-variables as well as their actions is below so you can use them to parameterize your
-installation of the software. Please note that all these variables have a default
-value which may not work well for you, but makes them all optional.
+TorPaste can be configured by using `ENV`ironment Variables. The list of
+available variables as well as their actions is below so you can use them to
+parameterize your installation of the software. Please note that all these
+variables have a default value which may not work well for you, but makes them
+all optional.
 
 ### Available ENV Variables
 
-* `TP_WEBSITE_TITLE` : Use this variable to set the TorPaste Title inside the HTML
-`<title></title>` tags. *Default:* `Tor Paste`
+* `TP_WEBSITE_TITLE` : Use this variable to set the TorPaste Title inside the
+HTML `<title></title>` tags. *Default:* `Tor Paste`
 * `TP_BACKEND` : Use this variable to select a backend for TorPaste to use. The
-available backends for each version are included in the `COMPATIBLE_BACKENDS` variable
-inside `torpaste.py`. *Default:* `filesystem`
-* `TP_PASTE_MAX_SIZE` : Use this variable to set the maximum paste size, in bytes. The
-possible values are formatted as `<amount> <unit>`, for example `10 M`, or `128 B`,
-or `16 k`. Any value that starts with `0` changes this limit to unlimited. *Default:*
-`0`
-* `TP_PASTE_LIST_ACTIVE` : Use this variable to enable or disable the paste listing
-available in the `Pastes` menu. *Default:* `True`
-* `TP_CSP_REPORT_URI` : Use this variable to set a `report-uri` for the Content Security
-Policy of TorPaste. If this variable is not set, no `report-uri` is added, which is the
-default behavior.
-* `TP_ENABLED_PASTE_VISIBILITIES` : Use this variable to select the available paste
-visibilities, separated by a comma. Example: "public,unlisted". The available backends
-for each version are included in the `AVAILABLE_VISIBILITIES` variable inside 
-`torpaste.py`. *Default:* `public`.
+available backends for each version are included in the `COMPATIBLE_BACKENDS`
+variable inside `torpaste.py`. *Default:* `filesystem`
+* `TP_PASTE_MAX_SIZE` : Use this variable to set the maximum paste size, in
+bytes. The possible values are formatted as `<amount> <unit>`, for example
+`10 M`, or `128 B`, or `16 k`. Any value that starts with `0` changes this
+limit to unlimited. *Default:* `0`
+* `TP_PASTE_LIST_ACTIVE` : Use this variable to enable or disable the paste
+listing available in the `Pastes` menu. *Default:* `True`
+* `TP_CSP_REPORT_URI` : Use this variable to set a `report-uri` for the Content
+Security Policy of TorPaste. If this variable is not set, no `report-uri` is
+added, which is the default behavior.
+* `TP_ENABLED_PASTE_VISIBILITIES` : Use this variable to select the available
+paste visibilities, separated by a comma. Example: "public,unlisted". The
+available backends for each version are included in the
+`AVAILABLE_VISIBILITIES` variable inside  `torpaste.py`. *Default:* `public`.
 
 ### Backend ENV Variables
-Each backend may need one or more additional `ENV` variables to work. For example,
-a MySQL backend may need the `HOST`, `PORT`, `USERNAME`, and `PASSWORD` to connect
-to the database. To prevent conflicts, all these variables will be available as
-`TP_BACKEND_BACKENDNAME_VARIABLE` where `BACKENDNAME` is the name of the backend,
-such as `MYSQL` and the `VARIABLE` will be the name of the variable, such as `HOST`.
-
-Currently there are no used backend `ENV` variables. When there are, you will find
-a list of all backends and their variables here.
+Each backend may need one or more additional `ENV` variables to work. For
+example, a MySQL backend may need the `HOST`, `PORT`, `USERNAME`, and
+`PASSWORD` to connect to the database. To prevent conflicts, all these
+variables will be available as `TP_BACKEND_BACKENDNAME_VARIABLE` where
+`BACKENDNAME` is the name of the backend, such as `MYSQL` and the `VARIABLE`
+will be the name of the variable, such as `HOST`.
+
+#### redis
+* `TP_BACKEND_REDIS_HOST` : The Redis server running to connect to. *Default:*
+`127.0.0.1`
+* `TP_BACKEND_REDIS_PORT` : The Redis server port to connect to. *Default:*
+`6379`
+* `TP_BACKEND_REDIS_PASSWORD` : The Redis server password needed for
+authentication. *Default: none*
+* `TP_BACKEND_REDIS_DB_INDEX` : The database index to connect to such as `0` or
+`1`. *Default:* `1`.

backends/redis.py

@@ -0,0 +1,183 @@
+#!../bin/python
+
+import backends.exceptions as e
+import redis
+from os import getenv
+
+
+def initialize_backend():
+    """
+    This method is called when the Flask application starts.
+    Here you can do any initialization that may be required,
+    such as connecting to a remote database or making sure a file exists.
+    """
+    global rb
+
+    RedisHost = getenv("TP_BACKEND_REDIS_HOST") or "127.0.0.1"
+    RedisPort = int(getenv("TP_BACKEND_REDIS_PORT") or "6379")
+    RedisPass = getenv("TP_BACKEND_REDIS_PASSWORD") or None
+    RedisDBI = int(getenv("TP_BACKEND_REDIS_DB_INDEX") or "1")
+
+    rb = redis.StrictRedis(
+        host=RedisHost,
+        port=RedisPort,
+        password=RedisPass,
+        db=RedisDBI
+    )
+
+    if rb.ping() is not True:
+        raise e.ErrorException(
+            "Could not connect to the backend database. Please try again" +
+            "later. If the error persists, please contact a system" +
+            "administrator."
+        )
+
+    rb.save()
+
+    return
+
+
+def new_paste(paste_id, paste_content):
+    """
+    This method is called when the Flask application wants to create a new
+    paste. The arguments given are the Paste ID, which is a paste identifier
+    that is not necessarily unique, as well as the paste content. Please note
+    that the paste must be retrievable given the above Paste ID as well as
+    he fact that paste_id is (typically) an ASCII string while paste_content
+    can (and will) contain UTF-8 characters.
+    :param paste_id: a not necessarily unique id of the paste
+    :param paste_content: content of the paste (utf-8 encoded)
+    :return:
+    """
+
+    rb.set(paste_id, paste_content)
+
+    rb.save()
+
+    return
+
+
+def update_paste_metadata(paste_id, metadata):
+    """
+    This method is called by the Flask application to update a paste's
+    metadata. For this to happen, the application passes the Paste ID,
+    which is typically an ASCII string, as well as a Python dictionary
+    that contains the new metadata. This method must overwrite any and
+    all metadata with the passed dictionary. For example, if a paste has
+    the keys a and b and this method is called with only keys b and c,
+    the final metadata must be b and c only, and not a.
+    :param paste_id: ASCII coded id of the paste
+    :param metadata: dictionary containing the metadata
+    :return:
+    """
+
+    for md in rb.keys(paste_id + ".*"):
+        rb.delete(md)
+    for md in metadata:
+        rb.set(paste_id + "." + md, metadata[md])
+
+    rb.save()
+
+    return
+
+
+def does_paste_exist(paste_id):
+    """
+    This method is called when the Flask application wants to check if a
+    paste with a given Paste ID exists. The Paste ID is (typically) an
+    ASCII string and your method must return True if a paste with this ID
+    exists, or False if it doesn't.
+    :param paste_id: ASCII string which represents the ID of the paste
+    :return: True if paste with given ID exists, false otherwise
+    """
+
+    return rb.exists(paste_id)
+
+
+def get_paste_contents(paste_id):
+    """
+    This method must return all the paste contents in UTF-8 encoding for
+    a given Paste ID. The Paste ID is typically in ASCII, and it is
+    guaranteed that this Paste ID exists.
+    :param paste_id: ASCII string which represents the ID of the paste
+    :return: the content of the paste in UTF-8 encoding
+    """
+
+    return rb.get(paste_id).decode("utf-8")
+
+
+def get_paste_metadata(paste_id):
+    """
+    This method must return a Python Dictionary with all the currently
+    stored metadata for the paste with the given Paste ID. All keys of
+    the dictionary are typically in ASCII, while all values are in
+    UTF-8. It is guaranteed that the Paste ID exists.
+    :param paste_id: ASCII string which represents the ID of the paste
+    :return: a dictionary with the metadata of a given paste
+    """
+
+    ret = {}
+
+    f = rb.keys(paste_id + ".*")
+    for md in f:
+        ret[md.split(".")[1]] = rb.get(md).decode("utf-8")
+
+    return str(ret)
+
+
+def get_paste_metadata_value(paste_id, key):
+    """
+    This method must return the value of the metadata key provided for
+    the paste whose Paste ID is provided. If the key is not set, the
+    method should return None. You can assume that a paste with this
+    Paste ID exists, and you can also assume that both parameters
+    passed are typically ASCII.
+    :param paste_id: ASCII string which represents the ID of the paste
+    :param key: key of the metadata
+    :return: value of the metadata key provided for the given ID, None if
+             the key wasn't set
+    """
+
+    if rb.exists(paste_id + "." + key) is not True:
+        return None
+    else:
+        return rb.get(paste_id + "." + key).decode("utf-8")
+
+
+def get_all_paste_ids(filters={}, fdefaults={}):
+    """
+    This method must return a Python list containing the ASCII ID of all
+    pastes which match the (optional) filters provided. The order does not
+    matter so it can be the same or it can be different every time this
+    function is called. In the case of no pastes, the method must return a
+    Python list with a single item, whose content must be equal to 'none'.
+    :param filters: a dictionary of filters
+    :param fdefaults: a dictionary with the default value for each filter
+                      if it's not present
+    :return: a list containing all paste IDs
+    """
+
+    ak = rb.keys()
+    ap = []
+    for k in ak:
+        if b"." not in k:
+            ap.append(k.decode("utf-8"))
+    filt = []
+    for p in ap:
+        keep = True
+
+        for k, v in filters.items():
+            gpmv = get_paste_metadata_value(p, k)
+            if gpmv is None:
+                gpmv = fdefaults[k]
+            if gpmv != v:
+                keep = False
+                break
+
+        if keep:
+            filt.append(p)
+
+    if len(filt) == 0:
+        filt = ['none']
+
+    return filt

requirements.txt

@@ -1 +1,2 @@
 Flask
+redis

torpaste.py

@@ -17,7 +17,7 @@
 VERSION = check_output(["git", "describe"]).decode("utf-8").replace("\n", "")
 
 # Compatible Backends List
-COMPATIBLE_BACKENDS = ["filesystem"]
+COMPATIBLE_BACKENDS = ["filesystem", "redis"]
 
 # Available list of paste visibilities
 # public: can be viewed by all, is listed in /list