Skip to content

Initial code for wikidata-based normalizer, untested #380

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 24 commits into
base: master
Choose a base branch
from
Open

Initial code for wikidata-based normalizer, untested #380

wants to merge 24 commits into from

Conversation

@nilesh-c
Copy link
Member

@nilesh-c nilesh-c commented Apr 16, 2015

Haven't tested this on sample data yet, mustn't merge. @chile12 could you check this?

@nilesh-c
Copy link
Member Author

nilesh-c commented Apr 16, 2015

Sorry, still got lots of build errors.

nilesh-c added 11 commits April 16, 2015 15:05
@nilesh-c
Copy link
Member Author

nilesh-c commented Apr 21, 2015

@chile12 could you test this out with a few languages? Maybe start up all the mappings langs with nohup. Here's the command I use:

../run NormalizeDatasets ../extraction.default.properties wikidata wikidata-sameas .ttl.bz2 page-links,infobox-properties -normalized

Specify your base-dir in the config file. wikidata is the language suffix for wikidatawiki, wikidata-sameas is the name of the dataset. Add all the datasets you want to normalize separated by commas.

jimkont
jimkont reviewed Apr 22, 2015
View reviewed changes
scripts/src/main/scala/org/dbpedia/extraction/scripts/NormalizeDatasets.scala Outdated
Copy link
Member

@jimkont jimkont Apr 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add wiki-code in the filename

Copy link
Member

@jimkont jimkont Apr 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the script is re-started with another language withouth deleting the file it will produce false results

Copy link
Member Author

@nilesh-c nilesh-c Apr 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you talking about the cache file? wikiFinder is instantiated with the baseDir and language - so for enwiki it creates the file enwiki-YYYYMMDD-sameas-mappings.obj in the respective directory.

Copy link
Member

@jimkont jimkont Apr 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perfect, you save them on the lang folder not the wikidata one, I got confused

@jimkont jimkont added this to the 2015-10 milestone Jun 1, 2015
@jimkont jimkont modified the milestones: 2015-10, 2016-04 Jul 25, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

2016-04

Development

Successfully merging this pull request may close these issues.

2 participants

@nilesh-c @jimkont