Wednesday, June 3, 2015

.NET Crawler Harvest

Alexander Nyquist ( ) made a cool crawler-component I used once... the solution worked quite good for me...

in my solution I wrote the page to the filesystem using the OnPageDownloaded and wrote some code for filtering items which redirect to a login page. Here a decision has to be made about using a white-list or a black-list approach. For full-text search purpose media content can be filtered too.


No comments: