How to evenly distribute data in apache pig output files? -


I have a dip-Latin script that takes XML in some, XPath uses UDF to remove some fields The resulting field stores: 

  Registry udf-lib-1.0-SNAPSHOT.jar; Define XPath com.blah.udfs.XPath (); Docs = Load '$ input' as com.blah.storage.XMLLoader ('root') (content: chararray); Results = FOREACH Documents Generation XPath (content, 'root / id'), XPath (content, 'root / other fields'), content; Store results in 'Output';

Note that we are using pig-0.12.0 on our cluster, so I split XPath / XMLLoader sections out of Dip-1 0.14 and in their own jar So that I can use them in 0.12.

The above script works fine and produces the data that I am looking for. However, it produces more than 1900 files with only a few MBs in each file. I have learned about the default_parallel option, so I tried from 128 and set to get 128 partfiles. I ended up adding a piece to reduce the phase to achieve it. My script now appears: 

  set default_parallel 128; Registrar udf-lib-1.0-SNAPSHOT.jar; Define XPath com.blah.udfs.XPath (); Docs = Load '$ input' as com.blah.storage.XMLLoader ('root') (content: chararray); Results = FOREACH Documents Generation XPath (content, 'root / id'), XPath (content, 'root / other fields'), content; Forced_readuce = Generate FOREACH (Group Results By Random) (Flatain) (Results); Compelled in '$ output';

Again, it produces the expected data. Besides, now I get 128 part-files. My problem is now that the data is not evenly distributed in the same-files In some 8 gigs, the other is 100 MB. I should remember this when grouped by Random () :.

My question is, what would be the preferred method of limiting the number of part-files, even then would they have the same size-size? I'm new in pig / dip Latin and assume I'm going completely wrong about it.

p.s Because I care about the number of part-files because I want to process production with Spark and our spark cluster works very well with very small files.

I'm still trying to do this with pag script directly, but now my " Solution "Re-dividing the data within Spark, the process that works on the production of pig scripts, I use the RDD.coalesce function to regulate the data.


Comments

Post a Comment

Popular posts from this blog

python - Overriding the save method in Django ModelForm -

I'm having trouble overriding a ModelForm save method. I get this error I'm: Exception type: TypeErier Exception Value: Save () found an unexpected keyword argument 'committed' My intentions include a form 3 fields Submit several values ​​for, then create an object for each combination of those areas, and to save each object file models.py class collarsult type (models.Model): id = model. AutoField (db_column = 'icontact_result_code_type_id', primary_key = true) callResult = models.ForeignKey ( 'CallResult', db_column = 'icontact_result_code_id') campaign = models.ForeignKey ( 'campaign', db_column = 'icampaign_id') CALLTYPE = models.ForeignKey ( 'CALLTYPE', db_column = 'icall_type_id') agent = models.BooleanField (db_column = 'bagent', default = true) teamLeader = models.BooleanField (db_column = 'bTeamLeader', default = true) active = models.BooleanField (db_column = Django.form...
Read more

html - CSS autoheight, but fit content to height of div -

I have a div in which there are three children- div in the left and middle div: float: left while true: float: true because it is my layout Actually will spoil, I've used ClearFix Hack: .cf {zoom: 1; }. Cf: First,. Cf: {content: ""; Display: Table; }. Cf: After {clear: both; } It still works, but I want to be the right div an indicator. So it should fill 100% of the parent unit of height. How can I complete this? PS is the full code: & Lt; / Div & gt; & Lt; Div class = "mobile_content" & gt; & Lt; H5 & gt; {{Data.titel}} & lt; / H5> & Lt; Table & gt; & Lt; TR & gt; & Lt; Th & gt; Komponist: & lt; / Th & gt; & Lt; TD & gt; {{Data.komponist}} & lt; / TD & gt; & Lt; / TR & gt; & Lt; TR & gt; & Lt; Th & gt; Instrument: & lt; / Th & gt; & Lt; TD & gt; {{Data.instrumente}} & lt; / TD & gt; & Lt; / TR & gt;...
Read more

qt - How to prevent QAudioInput from automatically boosting the master volume to 100%? -

I have been trying to use Qt5 multimedia to record audio with QAudioInput, however, when I see If QAudioInput is started, it increases the master volume of my sound device to 100%. How can I prevent QAudioInput from changing the master volume? My current development platform is Linux with PalsAdio (with flat audio disabled). How can I use QAudioInput : QAudioDeviceInfo device_info = QAudioDeviceInfo :: defaultInputDevice (); QAudio format format; Format.setSampleRate (44100); Format.setChannelCount (1); Format.setSampleSize (16); Format.setCodec ("Audio / PCM"); Format.setSampleType (QAudioFormat :: SignedInt); Format.setByteOrder (QAudioFormat :: LittleEndian); Std :: cout & lt; & Lt; Device_info.deviceName (). ToUtf8 (). ConstData () & lt; & Lt; Std :: endl; QAudioInput * default_device = new QAudioInput (device_info, format); QIODevice * default_io_device = default_device-> Start (); a QAudioInput.setVolume () method is not you I ...
Read more