Upgrade PyCSW

[adborden]

We're currently on a fork of 1.8.5 from Oct 2015. 2.2.0 is available from March 2018.

[adborden]

They're now on python 3, so we might consider separating the pycsw virtualenv from ckan.

[adborden]

Scratch that, it still works with python 2, but we should still put it in its own virtualenv.

[adborden]

I took a quick look at this and it's a low-medium effort. Our customizations to pycsw are all for the ckan loading, which exists in ckanext-spatial. So the plan becomes:

1. Pull in any relevant updates to ckanext-spatial.
2. Remove our fork of pycsw and use 2.2.0.
3. Reconfigure the load job to use ckan-pycsw from ckanext-spatial.

[adborden]

This is a blocker ckan/ckanext-spatial#49

[adborden]

But we can still upgrade our fork of pycsw.

[adborden]

https://github.com/geopython/pycsw/releases/tag/2.4.0 I don't see a changelog or similar, so I don't know exactly what the update brings.

[adborden]

Upgrading PyCSW to provide a more stable instance of CSW for FGDC.

[adborden]

In testing in staging, we'll need to upgrade requests and OWSLib. I don't know which specific versions, but the latest versions requests==2.22.0 and OWSLib==0.18.0 work in my testing. Need to check that the harvester still works.

Moving to the latest OWSLib, we should be able to drop our fork https://github.com/GSA/OWSLib

[adborden]

Just ran into this :/

2019-07-31 00:31:12,346 [pycsw-ckan] DEBUG: Fetching url=https://catalog-datagov.dev-ocsit.bsp.gsa.gov//harvest/object/68f3852b-53af-4953-b6d7-df9b67e48072
2019-07-31 00:31:12,378 [urllib3.connectionpool] DEBUG: https://catalog-datagov.dev-ocsit.bsp.gsa.gov:443 "GET //harvest/object/68f3852b-53af-4953-b6d7-df9b67e48072 HTTP/1.1" 200 3031
2019-07-31 00:31:12,379 [pycsw-ckan] DEBUG: Open Data JSON detected. Converting to ISO XML: de6c0602-115e-4d0b-8292-08d1e3ced8c0
Traceback (most recent call last):
  File "/usr/lib/ckan/bin/pycsw-ckan.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/usr/lib/ckan-new/src/pycsw/bin/pycsw-ckan.py", line 423, in <module>
    __reconcile(gathered_records, existing_records)
  File "/usr/lib/ckan-new/src/pycsw/bin/pycsw-ckan.py", line 281, in __reconcile
    record = get_record(ckan_api, CONTEXT, repo, CKAN_URL, ckan_id, ckan_info)
  File "/usr/lib/ckan-new/src/pycsw/bin/pycsw-ckan.py", line 143, in get_record
    template = env.get_template(tmpl)
  File "/usr/lib/ckan/lib/python2.7/site-packages/jinja2/environment.py", line 719, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/lib/ckan/lib/python2.7/site-packages/jinja2/environment.py", line 693, in _load_template
    template = self.loader.load(self, name, globals)
  File "/usr/lib/ckan/lib/python2.7/site-packages/jinja2/loaders.py", line 115, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "/usr/lib/ckan/lib/python2.7/site-packages/jinja2/loaders.py", line 180, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: datajson2iso.xml

[JJediny]

@kalxas @tomkralidis ?

Seems related to a no longer valid pycsw-ckan.py file that handles the import mapping from our outdated custom ckan/or/pycsw fork to the upgraded version of pycsw:

• https://github.com/PublicaMundi/ckanext-publicamundi/blob/master/ckanext/publicamundi/model/csw_record.py
• https://github.com/kindly/ckan_pycsw_import/blob/master/ckan_pycsw_import.py
• add support for CKAN integration geopython/pycsw#95

related to:

• PyCSW load fails to insert records due to field size for ix_records_othercontraints #606
• Database insert fails on large records geopython/pycsw#398

[adborden]

Work around by updating the path, it's using the source install path, not the virtualenv.

[adborden]

To update the database, there's no migration.

1. delete_records command is killed, likely OOM, so with psql, issue drop table commands.
2. Make sure postgis is enabled, CREATE EXTENSION postgis;
3. Create the tables $venv/bin/pycsw-ckan.py -c setup_db -f /etc/ckan/pycsw-all.cfg