Right now the extractor has to tell the crawler about desirable urls, but I think perhaps the crawler should watch what's in the graph to figure out where to go next.
For example, if a lot of concepts have relations to a particular url that hasn't been visited, it should go there next.
Right now the extractor has to tell the crawler about desirable urls, but I think perhaps the crawler should watch what's in the graph to figure out where to go next.
For example, if a lot of concepts have relations to a particular url that hasn't been visited, it should go there next.