Short Discussions don't work with JSON payloads

[deldesir]

The error raises wether I communicate to the bot by invoking the interactive mode with the --json (or -j) flag or running flask app accessible via a JSON endpoint.

---

ERROR in app: Exception on /reply [POST]
Traceback (most recent call last):
  File "/home/botmaster/.virtualenvs/a004/local/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/botmaster/.virtualenvs/a004/local/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/botmaster/.virtualenvs/a004/local/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/botmaster/.virtualenvs/a004/local/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/botmaster/.virtualenvs/a004/local/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "server.py", line 54, in reply
    reply = bot.reply(username, message)
  File "../../rivescript/rivescript.py", line 999, in reply
    return self._brain.reply(user, msg, errors_as_replies)
  File "../../rivescript/brain.py", line 69, in reply
    reply = self._getreply(user, msg, ignore_object_errors=errors_as_replies)
  File "../../rivescript/brain.py", line 301, in _getreply
    self.master.set_uservar(user, "__lastmatch__", matchedTrigger)
  File "../../rivescript/rivescript.py", line 757, in set_uservar
    self._session.set(user, {name: value})
  File "/home/botmaster/bot/eg/json-server/redis_storage.py", line 38, in set
    self.c.set(self.key(username), json.dumps(data))
  File "/usr/lib/python2.7/json/__init__.py", line 244, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <_sre.SRE_Pattern object at 0xb6b1ee58> is not JSON serializable

Edit/fixed the post formatting. --Kirsle

[htdinh]

@deldesir @kirsle Can you tell us more about how to reproduce this bug? Do we start from command line with the folder of the file shell.py and run:
python shell.py --json=...

[deldesir]

Yes, just start from terminal with echo '{"username": "kirsle", "message": "knock knock"}' | python shell.py --json eg/brain. The short discussion example used:

+ knock knock
- Who's there?

+ *
% who is there
- <star> who?

+ *
% * who
- LOL! <star>! That's funny!

[kirsle]

The original stack trace comes from the Redis session driver, so that would be a good thing to look into.

It sounds like somewhere in the user data structure, it's storing a data type that isn't JSON-serializable (looks like a regexp object). I don't know off-hand what that data could be; get_uservars() returns a normal dict of key/value string pairs of user data, plus the special structures like __history__ which should also be pretty simple (strings mainly).

To debug it I'd get as close as possible to where the exception is raised, and from pprint import pprint; pprint(data) to get a look at what the data contains.

[htdinh]

I try to the reproduce the issue but it doesn't throw me an error. I don't know which parts of the program require Flask/Redis session driver, as you mentioned. Can someone help explain further? Thank you.

Here is what do in my local machine. I prepare a script file test.rive at folder /test and put there the script of the short discussion. Where should the user variables be stored? Or it is erased whenever we stop the program?

+ knock knock
- Who is there?

+ *
% who is there
- <star> who?

+ *
% * who
- LOL! <star>! That's funny!

Below is the call and the response.

$ echo '{"username": "kirsle", "message": "knock knock"}'| python shell.py --json ../test

{'reply': "Who's there?",
 'status': 'ok',
 'vars': {'__history__': {'input': ['knock knock',
                                    'undefined',
                                    'undefined',
                                    'undefined',
                                    'undefined',
                                    'undefined',
                                    'undefined',
                                    'undefined',
                                    'undefined'],
                          'reply': ["Who's there?",
                                    'undefined',
                                    'undefined',
                                    'undefined',
                                    'undefined',
                                    'undefined',
                                    'undefined',
                                    'undefined',
                                    'undefined']},
          '__lastmatch__': 'knock knock',
          'topic': 'random'}}

[deldesir]

By default RiveScript stores all user variables and state in memory, the bug exists when using Redis cache to store user variables.

Here is my flask app as an example with Redis as session storage. Short discussions won't work with it as it would store something that is not JSON-serializable.

N.B.- I used redis_storage.py from eg/sessions/

#!/usr/bin/env python

# Manipulate sys.path to be able to import rivescript from this local git
# repository.
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), "..", ".."))

from flask import Flask, request, Response, jsonify
import json
from rivescript import RiveScript
from redis_storage import RedisSessionStorage

# Set up the RiveScript bot. This loads the replies from `/eg/brain` of the
# git repository.
bot = RiveScript(
    session_manager=RedisSessionStorage()
)
bot.load_directory(
    os.path.join(os.path.dirname(__file__), "..", "brain")
)
bot.sort_replies()

app = Flask(__name__)

@app.route("/reply", methods=["POST"])
def reply():
    """Fetch a reply from RiveScript.

    Parameters (JSON):
    * username
    * message
    * vars
    """
    params = request.json
    if not params:
        return jsonify({
            "status": "error",
            "error": "Request must be of the application/json type!",
        })

    username = params.get("username")
    message  = params.get("message")
#   uservars = params.get("vars", dict())

    # Make sure the required params are present.
    if username is None or message is None:
        return jsonify({
            "status": "error",
            "error": "username and message are required keys",
        })

    # Get a reply from the bot.
    reply = bot.reply(username, message)

    # Send the response.
    return reply 

@app.route("/")
@app.route("/<path:path>")
def index(path=None):
    """On all other routes, just return an example `curl` command."""
    payload = {
        "username": "soandso",
        "message": "Hello bot",
        "vars": {
            "name": "Soandso",
        }
    }
    return Response(r"""Usage: curl -i \
    -H "Content-Type: application/json" \
    -X POST -d '{}' \
    http://localhost:5000/reply""".format(json.dumps(payload)),
    mimetype="text/plain")

if __name__ == "__main__":
    app.run(host='0.0.0.0', debug=False)

[deldesir]

@Dinh-Hung-Tu curl -H "Content-Type: application/json" -X POST -d '{"message": "'"knock knock"'", "username": "'"kirsle"'"}' http://localhost:5000/reply reproduces the bug perfectly while having flask app and redis-server running.
Be sure to put debug to True (last line in flask app).
Available to give more details with this bug if necessary.

@kirsle How is the progress using from pprint import pprint; pprint(data)?