Proper etiquette for a web crawler http requests -


I have a simple web crawler to request all pages from a website's Sitemap, which I need to cache and index After many requests, the website starts serving blank pages.

There is nothing except their Sitemap link in their robots.txt , so I think I am not breaking their "rule" I have a descriptive header That really links to my intentions, and only from the sitemap of the pages that I crawl.

The http status codes are all still okay, so I can only imagine they are stopping it, in a short period of time a large number of http requests What is considered as a reasonable delay between requests?

Is there any other idea that I have ignored, which could potentially cause this problem?

Post text "itemprop =" text ">

Each site looks for different crawler and abuse features.

Simulate the crawler's key to human activity And robots.txt.

A detailed crawl will travel to some websites, and no matter how close they will be when you zip crawlers to some extent and one at a time. You suck everything. Generally you can get as much as 6 pages of pages

  • The following links will be safe in order of visibility on the webpage.
  • Try to ignore those links Do not show up on the webpage (many people use honey).

    • If everything fails, do not request faster than one page per minute If a website stops you at this rate, then contact them directly - Do not want to show that you use your content in this way.


    Comments

    Post a Comment

    Popular posts from this blog

    python - Overriding the save method in Django ModelForm -

    I'm having trouble overriding a ModelForm save method. I get this error I'm: Exception type: TypeErier Exception Value: Save () found an unexpected keyword argument 'committed' My intentions include a form 3 fields Submit several values ​​for, then create an object for each combination of those areas, and to save each object file models.py class collarsult type (models.Model): id = model. AutoField (db_column = 'icontact_result_code_type_id', primary_key = true) callResult = models.ForeignKey ( 'CallResult', db_column = 'icontact_result_code_id') campaign = models.ForeignKey ( 'campaign', db_column = 'icampaign_id') CALLTYPE = models.ForeignKey ( 'CALLTYPE', db_column = 'icall_type_id') agent = models.BooleanField (db_column = 'bagent', default = true) teamLeader = models.BooleanField (db_column = 'bTeamLeader', default = true) active = models.BooleanField (db_column = Django.form...
    Read more

    html - CSS autoheight, but fit content to height of div -

    I have a div in which there are three children- div in the left and middle div: float: left while true: float: true because it is my layout Actually will spoil, I've used ClearFix Hack: .cf {zoom: 1; }. Cf: First,. Cf: {content: ""; Display: Table; }. Cf: After {clear: both; } It still works, but I want to be the right div an indicator. So it should fill 100% of the parent unit of height. How can I complete this? PS is the full code: & Lt; / Div & gt; & Lt; Div class = "mobile_content" & gt; & Lt; H5 & gt; {{Data.titel}} & lt; / H5> & Lt; Table & gt; & Lt; TR & gt; & Lt; Th & gt; Komponist: & lt; / Th & gt; & Lt; TD & gt; {{Data.komponist}} & lt; / TD & gt; & Lt; / TR & gt; & Lt; TR & gt; & Lt; Th & gt; Instrument: & lt; / Th & gt; & Lt; TD & gt; {{Data.instrumente}} & lt; / TD & gt; & Lt; / TR & gt;...
    Read more

    qt - How to prevent QAudioInput from automatically boosting the master volume to 100%? -

    I have been trying to use Qt5 multimedia to record audio with QAudioInput, however, when I see If QAudioInput is started, it increases the master volume of my sound device to 100%. How can I prevent QAudioInput from changing the master volume? My current development platform is Linux with PalsAdio (with flat audio disabled). How can I use QAudioInput : QAudioDeviceInfo device_info = QAudioDeviceInfo :: defaultInputDevice (); QAudio format format; Format.setSampleRate (44100); Format.setChannelCount (1); Format.setSampleSize (16); Format.setCodec ("Audio / PCM"); Format.setSampleType (QAudioFormat :: SignedInt); Format.setByteOrder (QAudioFormat :: LittleEndian); Std :: cout & lt; & Lt; Device_info.deviceName (). ToUtf8 (). ConstData () & lt; & Lt; Std :: endl; QAudioInput * default_device = new QAudioInput (device_info, format); QIODevice * default_io_device = default_device-> Start (); a QAudioInput.setVolume () method is not you I ...
    Read more