question on crawler #24

Activity

1. polyrabbit commented on Jun 15, 2023

   

   polyrabbit

   on Jun 15, 2023

   Owner

   More actions

   Actually, I started this project almost 9 years ago - late 2014 (see my first commit), when there are only few open-sourced extractors, and they didn't perform well at that time.

   One reason to write it from scratch is flexibility and customizability - I can tune the parameters so that it suits better for HN posts. One case is the HN comments page, it appears frequently on front-page but most extractors do not get the right content.

   I'll try some of the modern ones later, thanks.
2. thiswillbeyourgithub commented on Jun 15, 2023

   

   thiswillbeyourgithub

   on Jun 15, 2023

   Author

   More actions

   Interesting thanks.