Investigate possible approaches and level of effort to calculate a GoldRush key for each MARC record

[corylown]

Deliverable is documenting findings and recommended approach on this issue.

See: #1167

[corylown]

We would also like to get a sense for how much this slows down the normalization process.

[JohnMarkOckerbloom]

I will write to my contacts to see if there is already benchmark data on how much time it takes to calculate a Gold Rush key per record.

[JohnMarkOckerbloom]

I've heard back from George Machovec at the Colorado Alliance of Research Libraries. He writes

"Our matchkey code is written in Java and creates matchkeys at around 4,000 per minute which translates to 5-6 million per day. It actually probably runs faster than that depending on how it is incorporated in your broader framework.

We would be willing to share the Java code with you if there is interest down the road."

[JohnMarkOckerbloom]

I've also heard back from Ian Bogus, who reports a notably faster performance with a Python implementation:

"A few years ago ReCAP was looking at different algorithms. I have a version written in Python that makes a CSV that has the key. I ran it and it generated about 3.3 million keys from the same number of records in about 2.5 hours. Granted, this is a stand-alone script and doesn’t include indexing or anything."

[dnoneill]

John is going to double check that is okay to put this functionality in the analytics application. Team recommendation based on performance/code bases is to have this functionality in analytics application assuming there are no reasons to put in the aggregator.