mrjob has a lot of options that are designed to upload a file, run a command, or add something to the environment:

• bootstrap_actions
• bootstrap_cmds
• bootstrap_files
• bootstrap_mrjob
• bootstrap_python_packages
• bootstrap_scripts
• file_upload_args (not available from the command line)
• python_archives
• setup_cmds
• setup_scripts
• upload_archives
• upload_files

There are two main problems with the way things are now:

• It's confusing
• Options don't always run in the order you want. For example, if you upgrade Python in bootstrap_cmds, bootstrap_python_packages becomes useless because it runs first (so it'll install packages for the old version of Python).

I don't have a complete solution, but I imagine something where you simply specify the commands you want to run, possibly referencing local paths or S3/HDFS URIs, and mrjob just does the right thing. We just need a clean way of disambiguating local and remote files.

We should aim to make this solution the canonical way of doing things in mrjob v0.4.

The goal of this ticket is a full port of mrjob that works for everything except EMR. This includes tests.

Roadmap:

• v0.4.2: Last release to support 2.5
• v0.4.4: Experimental support for Python 3, except EMR
• v0.5.0: Full support for Python 3.

This does not include EMR support (#989) or updating docs and examples (#994).

This implements the initial support for specifying the input and output for the jar. It uses INPUT_MARKER and OUTPUT_MARKER. It is still probably suboptimal and needs tests. Let me know what you think.

This commit adds support for starting an emr job flow with spot instances. A sample mrjob.conf follows.

Please note: this commit depends on the master branch from:

https://github.com/hblanks/boto

A pull request for this branch is out to the main boto repo, at https://github.com/boto/boto/pull/322.

This commit makes fairly small changes but does demand a certain amount of YAML'ing in order to specify instance groups. If there's anything I can be doing better, please let me know.

Sample mrjob.conf:

runners:
    emr:
        emr_instance_groups:
            -
                count: 1
                role: MASTER
                instance_type: m1.small
                market: SPOT
                name: "[email protected]"
                bid_price: "0.20"

            -
                count: 3
                role: CORE
                instance_type: c1.medium
                market: SPOT
                name: "[email protected]"
                bid_price: "0.20"

Currently, using -libjars requires something like this:

--bootstrap-file myjar.jar --bootstrap-cmd 'cp myjar.jar /home/hadoop/myjar.jar' --hadoop-arg -libjars --hadoop-arg /home/hadoop/myjar.jar

It would be nice if we could do this instead:

--libjar myjar.jar

mrjob would:

• Upload the file like the other bootstrap files
• Copy it to a unique location like /tmp/myjar-235324.jar
• Add the appropriate -libjars argument to StreamingStep.step_args

Migrating from dumbo (the other MapReduce Python module) should be pretty easy because its mappers and reducers have the same function signature.

Would be great to have some input from someone who actually uses dumbo so we're not just making stuff up. :)

For 2.x & 0.23, mkdir need -p for creating parent directories:

http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/FileSystemShell.html#mkdir http://hadoop.apache.org/docs/r0.23.10/hadoop-project-dist/hadoop-common/FileSystemShell.html#mkdir

For 1.x & 0.1x & 0.20, mkdir doesn't need any additional parameters: http://hadoop.apache.org/docs/r1.2.1/file_system_shell.html#mkdir

This would help with #753 and other attempts to support binary data.

Say you want to take the US Census TIGER Shapefiles as the input to the first step of your job. You want each mapper to receive exactly one file, and you need to know each file's name so you can match up the .shp and .dbf files in your reducer. And you don't want to write any Java, or include any custom jars.

I think mrjob could handle it this way:

• make a manifest file, with the URI of one input file per line.
• use --jobconf mapred.max.split.size=1 to force Hadoop to pass one line to each mapper.
• in the wrapper script that we already use for setup commands:
  • read the URI from stdin
  • export jobconf environment variables (e.g. map_input_file) so that the job thinks it's reading directly from the file.
  • use hadoop fs to pipe the file into the wrapped command: set -o pipefail; hadoop fs -cat <uri> | "$@"
• The job reads directly from self.stdin in its mapper_init() step (see #753).

All local and inline modes have to do to emulate this is to not split input files.

Here is a patch to add support for typedbytes to mrjob.

To enable a class to use typedbytes instead of the textual interface, you just need to specify

class MyMRJob(MRJob): STREAMING_INTERFACE = MRJob.STREAMING_INTERFACE_TYPED_BYTES

And then the interface to hadoop streaming will use typedbytes instead of line based formats.

This is compatible with dumbo, and adds typedbytes as a requirement to MRJob.

This fixes #430.

Issue for pig support opened in https://github.com/Yelp/mrjob/issues/377

I also have a few other patches with this pull request - https://github.com/Yelp/mrjob/issues/317 and the ability to force clear the output folder, --force-clear-output-dir. The latter is used only for debugging jobs.

I need to add integration tests for these. I delayed this for some time due to the fear of testify, but now can add these in soon.

This is an early version especially for the pig support. Feel free to provide me with suggestions or coding style. I am relatively new to python but am using it frequently off late.

Thanks, Shiv

s3n:// is the technically correct way for EMR to refer to files on S3, but nowadays, s3:// does the exact same thing.

(s3:// used to refer to some legacy block format that only HDFS could use; now it's deprecated, and you have to use s3bfs:// instead.)

We should probably allow users to use s3:// or s3n:// as they prefer, and translate either to s3:// for consistency.

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

There are small typos in:

• docs/whats-new.rst
• mrjob/examples/mr_text_classifier.py
• mrjob/sim.py

Fixes:

• Should read refers rather than referse.
• Should read consistently rather than consistenly.
• Should read because rather than becaue.

Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

I set GOOGLE_APPLICATION_CREDENTIALS env variable properly, and am running a simple mrjob with -r dataproc option. However, it says

google.api_core.exceptions.Unknown: None Stream removed

While calling

self.cluster_client.get_cluster()

I see that the v1beta2 API has been deprecated in favor of v1. Does the dataproc plugin need a re-write to adopt it? Anything I can do to help?

We are using MrJob to process WARC files, in similar manner to this example given in the Writing Jobs guide.

For our use case, it is crucial that the .gz compressed file is not automatically decompressed before use.

This PR proposes a new setting that would allow this to be controlled via a unpack_archives option passed to the MrJob runner. This new option defaults to True to maintain the expected default behaviour, while allowing us to set it to False when needed. We have tested this locally and it seems to work just fine.

I've attempted to document this new option, as per the contributing guidelines, but I'm not sure I've covered everything. Is there any other documentation I should add?