There seems to be a weird interaction when you run jug cleanup while jobs are still running, but I could be wrong, but in addition, when a job is ended by sigterm jobs seem to enter the completed stage and never finish.

A light-weight wrapper to run jug with gridmap on a Grid Engine cluster

Get the best of both worlds of gridmap and jug

• http://pythonhosted.org/Jug/
• http://gridmap.readthedocs.org/en/latest/

This implements jug.grid_jug to run jug jobs with all the comfortable features of gridmap.

As I am not sure whether this belongs into the gridmap or the jug package, I create a pull request for both to foster discussion between the two projects. The file jug/grid.py is identical with the file gridmap/jug.py .

When using subprocess.Popen(["jug", "execute", "exmp.py"], stderr=STDOUT, creationflags=CREATE_NEW_CONSOLE) runing jug process, the subprocess will only starts after checking status. The same thing happens when running by subprocess.Popen(["jug", "execute", "exmp.py"], stderr=STDOUT, shell=True)

This problem doesn't appear when using subprocess.Popen(["jug", "execute", "exmp.py"], stderr=STDOUT)

Would you mind to help?

Hi, I wrote a small piece of python code on a macbook which has 10 + 1 tasks. 10 similar sleeping task and 1 joint task. I am running it by 4 processes and found that sometime Jug cannot assign task correctly. It happens like one process does all tasks along and, at the same time, another 3 share the 11 tasks.

If jugdir is not provided on command-line to "jug execute", and there is no jug config file, the program errors out, since this is prior to getting to the call to init() which handles the missing jugdir.

Just define a default:

parser.add_option('--jugdir',
                action='store',
                dest='jugdir',
                default='jugdir', # define default value
                help='Where to save intermediate results')

I have been using a decorator to provide the functionality of iteratetask instead of the function.

The reason I like this is that it specifies how to handle the output (i.e. how many items in the tuple) closer to the place the output is specified (i.e. at the end of the function) rather than where the workflow is specified (which is usually a different file).

So it looks like:

@IterateTask(n=3)
@jug.TaskGenerator
def do_thing(args):
    return a, b, c

I've been using the following for a few weeks without incident:

import jug
from functools import wraps

def IterateTask(n):
    def decorator(f):
         @wraps(f)
         def wrapper(*args, **kwargs):
             return jug.iteratetask(f(*args, **kwargs), n=n)
         return wrapper
    return decorator

Is this something you'd be interested in adding? Is there any reason I haven't foreseen that this isn't a good idea?

Hi, I defined a class which has a init. There are some methods in this class is decorated with @TaskGenerator. These methods cannot get class instance correctly. I tried 2 ways to solve it and they all don't work.

1. decorate init with @TaskGenerator then init() return's a jug Task instead of None
2. put a barrier() after defining the class object then got TypeError: cannot serialize '_io.TextIOWrapper' object

Would you mind to tell me the correct way to do it?

sys.argv now contains jugfile and unparsed arguments (i.e. after --) The subcommand api tests were also leaking causing other parts of the testsuite to fail.

Came across this error today:

CRITICAL:root:Exception while running tangent_lstm.test_lstm: maximum recursion depth exceeded while calling a Python object
Traceback (most recent call last):
  File "/gpfs/main/home/skainswo/Research/kaggle_gal/venv/bin/jug", line 5, in <module>
    main()
  File "/gpfs/main/home/skainswo/Research/kaggle_gal/venv/local/lib/python2.7/site-packages/jug/jug.py", line 415, in main
    execute(options)
  File "/gpfs/main/home/skainswo/Research/kaggle_gal/venv/local/lib/python2.7/site-packages/jug/jug.py", line 254, in execute
    execution_loop(tasks, options)
  File "/gpfs/main/home/skainswo/Research/kaggle_gal/venv/local/lib/python2.7/site-packages/jug/jug.py", line 163, in execution_loop
    t.run(debug_mode=options.debug)
  File "/gpfs/main/home/skainswo/Research/kaggle_gal/venv/local/lib/python2.7/site-packages/jug/task.py", line 95, in run
    self.store.dump(self._result, name)
  File "/gpfs/main/home/skainswo/Research/kaggle_gal/venv/local/lib/python2.7/site-packages/jug/backends/file_store.py", line 139, in dump
    encode_to(object, output)
  File "/gpfs/main/home/skainswo/Research/kaggle_gal/venv/local/lib/python2.7/site-packages/jug/backends/encode.py", line 77, in encode_to
    write(object, stream)
RuntimeError: maximum recursion depth exceeded while calling a Python object

I'm getting this error when creating a Task that wraps a function which returns a Keras model.

• The url decomposition regexp for the redis back-end apparently had a problem.
• The hash-from-arguments approach didn't work for my particular problem: a type can be 'pickle-able' without the pickle being unique, something that makes tasks non-uniquely identifiable. So, made it possible for the user to control the hash somehow.

I recently had an issue with a few of my jug processes being killed and therefore locks being kept on some active tasks. Normally, I would just delete the lock files in the jugdir, but I was using the Redis backend, so this was not possible. As suggested in the docs, I ran jug cleanup, expecting only the locks to be cleared; I was surprised to find the completed jobs cleared as well.

Is there any way to have jug cleanup only clear the locks? If not, can there be an option for this?

-------------------------------------------------------------------------------------
           0           0          1           0           0  jugfile.function
           0           0          1           0           0  jugfile.function_other
.....................................................................................
           0           0          2           2           0  Total

Running jug invalidate --target jugfile.function invalidates everything.