python method to extract content (excluding navigation) from an HTML page -


Of course, an HTML page can be parsed using any number of python parser, but I'm afraid Not to seem to have a public parsing script to extract meaningful content (excluding sidebar, navigation, etc.) from the given HTML document

I'm guessing it like collecting the DIV and P elements And then for the minimum content of them I'll check it, but I'm convinced that a solid implementation will include a lot of things I have not thought that

Try the library for Python. There are very easy ways to extract information from an HTML file.

Trying to extract data normally from the webpages will require people to type their pages in the same way ... but in this almost unlimited number of methods express a page that gives the same information All you can have to express is equally identical.

Were you trying to extract a specific kind of information or some other end goal?

You can try to remove any material in the 'div' and 'p' markers and compare the relative size of all the information in the page. The problem occurs when people send group information to the collection of 'div' and 'p' (or at least they do this, they are writing the well-formed HTML!). Maybe << p>

Information is related (nodes' P 'or' div or whatever and all the text contained in the node ') You can do some kind of analysis, which can be identified by the smallest' P 'or' Div 'that most encapses to be included in it ..?

[edit] Perhaps if you can get it in the tree structure, then I suggested that you can use the same number system for the spam killer. . Define some rules that try to classify the information. Some examples: 

  +1 points for every 100 words + 1 point for each child element & gt; 100 words -1 points if the section name contains the word 'NAV' -2, if the word 'advert' is in the section name then

If you have very few scoring rules that add When you find a more popular looking category, then I think it can grow into a very powerful and strong technique.

[EDIT2] Looking at the readability, it feels great that it is okay what I just suggested! Probably can improve tables and improve understanding?


Comments

Post a Comment

Popular posts from this blog

python - Overriding the save method in Django ModelForm -

I'm having trouble overriding a ModelForm save method. I get this error I'm: Exception type: TypeErier Exception Value: Save () found an unexpected keyword argument 'committed' My intentions include a form 3 fields Submit several values ​​for, then create an object for each combination of those areas, and to save each object file models.py class collarsult type (models.Model): id = model. AutoField (db_column = 'icontact_result_code_type_id', primary_key = true) callResult = models.ForeignKey ( 'CallResult', db_column = 'icontact_result_code_id') campaign = models.ForeignKey ( 'campaign', db_column = 'icampaign_id') CALLTYPE = models.ForeignKey ( 'CALLTYPE', db_column = 'icall_type_id') agent = models.BooleanField (db_column = 'bagent', default = true) teamLeader = models.BooleanField (db_column = 'bTeamLeader', default = true) active = models.BooleanField (db_column = Django.form...
Read more

html - CSS autoheight, but fit content to height of div -

I have a div in which there are three children- div in the left and middle div: float: left while true: float: true because it is my layout Actually will spoil, I've used ClearFix Hack: .cf {zoom: 1; }. Cf: First,. Cf: {content: ""; Display: Table; }. Cf: After {clear: both; } It still works, but I want to be the right div an indicator. So it should fill 100% of the parent unit of height. How can I complete this? PS is the full code: & Lt; / Div & gt; & Lt; Div class = "mobile_content" & gt; & Lt; H5 & gt; {{Data.titel}} & lt; / H5> & Lt; Table & gt; & Lt; TR & gt; & Lt; Th & gt; Komponist: & lt; / Th & gt; & Lt; TD & gt; {{Data.komponist}} & lt; / TD & gt; & Lt; / TR & gt; & Lt; TR & gt; & Lt; Th & gt; Instrument: & lt; / Th & gt; & Lt; TD & gt; {{Data.instrumente}} & lt; / TD & gt; & Lt; / TR & gt;...
Read more

qt - How to prevent QAudioInput from automatically boosting the master volume to 100%? -

I have been trying to use Qt5 multimedia to record audio with QAudioInput, however, when I see If QAudioInput is started, it increases the master volume of my sound device to 100%. How can I prevent QAudioInput from changing the master volume? My current development platform is Linux with PalsAdio (with flat audio disabled). How can I use QAudioInput : QAudioDeviceInfo device_info = QAudioDeviceInfo :: defaultInputDevice (); QAudio format format; Format.setSampleRate (44100); Format.setChannelCount (1); Format.setSampleSize (16); Format.setCodec ("Audio / PCM"); Format.setSampleType (QAudioFormat :: SignedInt); Format.setByteOrder (QAudioFormat :: LittleEndian); Std :: cout & lt; & Lt; Device_info.deviceName (). ToUtf8 (). ConstData () & lt; & Lt; Std :: endl; QAudioInput * default_device = new QAudioInput (device_info, format); QIODevice * default_io_device = default_device-> Start (); a QAudioInput.setVolume () method is not you I ...
Read more