A different dulwich error using Fedora 23

I've generated a 64-bit Fedora 23 workstation virtual machine to try (among other reasons) to get closer to what other ofCourse developers may be using in their own builds.

I end up with this traceback:

Traceback (most recent call last):
  File "/home/joe/hfoss-S16/ofcourse_environment/bin/ofcourse", line 10, in 
    sys.exit(cli())
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/ofcourse/cli/__init__.py", line 204, in openshift
    app_url = push(appname, api, domain)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/ofcourse/cli/openshift_utils.py", line 119, in push
    map(os.remove, openshift_files.keys())
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/ofcourse/cli/openshift_utils.py", line 79, in __exit__
    six.reraise(exc_type, value, tb)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/ofcourse/cli/openshift_utils.py", line 101, in push
    repo.do_commit("Commit openshift files")
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/dulwich/repo.py", line 565, in do_commit
    committer = self._get_user_identity()
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/dulwich/repo.py", line 496, in _get_user_identity
    config.get(("user", ), "name"),
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/dulwich/config.py", line 380, in get
    raise KeyError(name)
KeyError: 'name'

To better document and make consistent what I'm doing, I made the attached script to run through the whole process of generating a test course, and captured a run of it in the second attached file using 'script'.

script.txt
output.txt

[ryansb]

It seems like your gitconfig lacks a user/name. Try git config --global user.name "Dee Joe"

[ryansb]

This is due to a missing git username. Reopen if that doesn't fix it.

I'm getting this stack trace

Traceback (most recent call last):
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/bin/ofcourse", line 11, in <module>
    sys.exit(cli())
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/__init__.py", line 159, in openshift
    if is_dirty():
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/openshift_utils.py", line 130, in is_dirty
    s = git.status(repo)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/porcelain.py", line 549, in status
    tracked_changes = get_tree_changes(r)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/porcelain.py", line 564, in get_tree_changes
    index = r.open_index()
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/repo.py", line 725, in open_index
    return Index(self.index_path())
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/index.py", line 203, in __init__
    self.read()
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/index.py", line 223, in read
    f = SHA1Reader(f)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/pack.py", line 1360, in __init__
    self.sha1 = sha1('')
TypeError: Unicode-objects must be encoded before hashing

when running any one of several variants of this script:

GHACCT=ritjoe
COURSE_NAME=testofcourse
SEMESTER=F16
COURSEDIR=${COURSE_NAME}-${SEMESTER}
GHURL=https://github.com/$GHACCT/$COURSE_NAME
#USERNAME="D Joe"
#EMAIL="deejoe+artemisia@etrumeus.com"

mkdir $COURSEDIR
cd $COURSEDIR

virtualenv --no-site-packages -p python3 ofcourse_environment
source ofcourse_environment/bin/activate
pip install ofcourse
mkdir $COURSE_NAME

cd $COURSE_NAME
ofcourse new
git init
git config --replace-all --global user.name "Joe"
git config --replace-all --global user.email "ritjoe@etrumeus.com"
git add .
git commit -m "initial commit"
ofcourse openshift --app $COURSE_NAME --domain ritjoe

It's working up through the openshift deployment.

As per #75 (comment) I've played around with how I quote user.name and user.email but it seems not to matter.

I'm not sure I can re-open this myself:

http://stackoverflow.com/a/21333938

[ryansb]

That appears to be a dulwich bug - it's not resulting from things ofcourse passes in, but rather choking on decoding the sha (inside plumbing).

pip freeze output, as requested via IRC

$ pip freeze
ansible==2.1.1.0
Babel==2.3.4
ecdsa==0.13
httplib2==0.9.2
iniparse==0.4
Jinja2==2.8
MarkupSafe==0.23
paramiko==1.16.0
pyasn1==0.1.9
pycrypto==2.6.1
pygpgme==0.3
pyliblzma==0.5.3
python-keyczar==0.71rc0
pytz==2016.6.1
PyYAML==3.11
rpm-python==4.13.0rc1
six==1.10.0
virtualenv==14.0.6
ESC[33mYou are using pip version 8.0.2, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.ESC[0m

[ryansb]

dulwich isn't in there - is it living outside your virtualenv or something?

Not sure when the script bombs out of the venv, but making sure I'm in the venv and trying the ofcourse openshift step again interactively I get what looks like the same trace, but with this pip freeze:

ESCkjoe@localhost:~/scripted-ofcourse/testofcourse-F16/testofcourseESC\(ofcourse_environment) [joe@localhost testofcourse]$ pip freeze
click==6.6
dulwich==0.10.0
feedparser==5.2.1
Flask==0.11.1
Flask-Mako==0.4
Frozen-Flask==0.12
icalendar==3.10
itsdangerous==0.24
Jinja2==2.8
Mako==1.0.4
MarkupSafe==0.23
ofcourse==0.2.5
oshift==0.2.3
python-dateutil==2.5.3
pytz==2016.6.1
PyYAML==3.11
requests==2.11.1
six==1.10.0
tornado==4.4.1
validator.py==1.2.5
Werkzeug==0.11.10

Confirmed that it was just that the subshell from my script that was in the venv and that therefor left the venv as the subshell exited on finishing the script.

So, that pip freeze is representative of what is in the venv

issuing

pip install --upgrade dulwich

from within the venv takes dulwich from 0.10.0 to 0.14.1 which moves us past the error above, but then generates

Traceback (most recent call last):
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/bin/ofcourse", line 11, in <module>
    sys.exit(cli())
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/__init__.py", line 159, in openshift
    if is_dirty():
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/openshift_utils.py", line 131, in is_dirty
    return any(s.staged.values() + [s.unstaged])
TypeError: unsupported operand type(s) for +: 'dict_values' and 'list'

That can then be ameliorated by making the line in question:

    return any([s.staged.values()] + [s.unstaged])

Allows things to move a bit further, before running into this error:

Traceback (most recent call last):
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/bin/ofcourse", line 11, in <module>
    sys.exit(cli())
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/__init__.py", line 204, in openshift
    app_url = push(appname, api, domain)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/openshift_utils.py", line 93, in push
    git.reset(repo, "hard")
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/porcelain.py", line 558, in reset
    tree = r[committish].tree
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/repo.py", line 449, in __getitem__
    type(name).__name__)
TypeError: 'name' must be bytestring, not str

and that's as far as I've gotten.

It looks like they've been working on this in dulwich, see this latest commit: Fix python3 compatibility

But I think the most recent release (from early July) does not yet include the python3 fix

More from our IRC conversation, with regard to forcing an even more recent version of Dulwich with python3 fixes (see above), pending its release:

You can use the git URL in your requirements file
to get the dulwich devel version with Py3 fixes
locally you can do 
pip install -U --force  'git+https://github.com/jelmer/dulwich.git#egg=dulwich'

[ryansb]

Great, that looks like the best solution for now. Once a new release is out, I'll pin the requirements.txt to the newer version for Python 3 installations.