storage - Implementing large scale log file analytics -


Does anyone point me in any context or provide a high level overview of how Facebook, Yahoo, Google etc. Companies exhibit large scale (for example, multi-TB class) log analysis that they do for operations and specifically web analytics?

Especially focusing on web analytics, I am interested in two related aspects: query execution and data collection.

I know that using common approaches to reduce the map, distribute every query on the cluster (like using Hadoop) However, what is the most efficient storage format to use? This is log data, so we can assume that each event has a time stamp, and in general the data is structured and not sparse. In most web analytics queries, the slides of the data have to be analyzed between two arbitrary timestamps, and that data involves retrieving the total data or discrepancies.

Will a column-oriented DB be an effective way to store big tables (or HBs), and more importantly, such data queries? Is it the fact that you are selecting a subset of rows (based on timestamps) working against the basic premise of this type of storage? Would it be better to store it as unstructured data, such as a reverse index?

Unfortunately any size fits in all answers.

I am currently using Cascading, Hadoop, S3 and Astor data to process 100 days of gigas through a phased pipeline inside the SWS.

Esther data is used for query and reporting because it provides a SQL interface clears large data and is parsed by Cascading Processes at Hadoop using Cascading JDBC Interface, Loading ester data is a trivial process.

Keep key / value stores like Hibbes and HyperTable, so do not do ad-hoc queries and join without help

In full disclosure, I am a developer on the cascading project.


Comments

Post a Comment

Popular posts from this blog

python - Overriding the save method in Django ModelForm -

I'm having trouble overriding a ModelForm save method. I get this error I'm: Exception type: TypeErier Exception Value: Save () found an unexpected keyword argument 'committed' My intentions include a form 3 fields Submit several values ​​for, then create an object for each combination of those areas, and to save each object file models.py class collarsult type (models.Model): id = model. AutoField (db_column = 'icontact_result_code_type_id', primary_key = true) callResult = models.ForeignKey ( 'CallResult', db_column = 'icontact_result_code_id') campaign = models.ForeignKey ( 'campaign', db_column = 'icampaign_id') CALLTYPE = models.ForeignKey ( 'CALLTYPE', db_column = 'icall_type_id') agent = models.BooleanField (db_column = 'bagent', default = true) teamLeader = models.BooleanField (db_column = 'bTeamLeader', default = true) active = models.BooleanField (db_column = Django.form...
Read more

html - CSS autoheight, but fit content to height of div -

I have a div in which there are three children- div in the left and middle div: float: left while true: float: true because it is my layout Actually will spoil, I've used ClearFix Hack: .cf {zoom: 1; }. Cf: First,. Cf: {content: ""; Display: Table; }. Cf: After {clear: both; } It still works, but I want to be the right div an indicator. So it should fill 100% of the parent unit of height. How can I complete this? PS is the full code: & Lt; / Div & gt; & Lt; Div class = "mobile_content" & gt; & Lt; H5 & gt; {{Data.titel}} & lt; / H5> & Lt; Table & gt; & Lt; TR & gt; & Lt; Th & gt; Komponist: & lt; / Th & gt; & Lt; TD & gt; {{Data.komponist}} & lt; / TD & gt; & Lt; / TR & gt; & Lt; TR & gt; & Lt; Th & gt; Instrument: & lt; / Th & gt; & Lt; TD & gt; {{Data.instrumente}} & lt; / TD & gt; & Lt; / TR & gt;...
Read more

qt - How to prevent QAudioInput from automatically boosting the master volume to 100%? -

I have been trying to use Qt5 multimedia to record audio with QAudioInput, however, when I see If QAudioInput is started, it increases the master volume of my sound device to 100%. How can I prevent QAudioInput from changing the master volume? My current development platform is Linux with PalsAdio (with flat audio disabled). How can I use QAudioInput : QAudioDeviceInfo device_info = QAudioDeviceInfo :: defaultInputDevice (); QAudio format format; Format.setSampleRate (44100); Format.setChannelCount (1); Format.setSampleSize (16); Format.setCodec ("Audio / PCM"); Format.setSampleType (QAudioFormat :: SignedInt); Format.setByteOrder (QAudioFormat :: LittleEndian); Std :: cout & lt; & Lt; Device_info.deviceName (). ToUtf8 (). ConstData () & lt; & Lt; Std :: endl; QAudioInput * default_device = new QAudioInput (device_info, format); QIODevice * default_io_device = default_device-> Start (); a QAudioInput.setVolume () method is not you I ...
Read more