Output of pip install scoop:

Collecting scoop
  Could not find a version that satisfies the requirement scoop (from versions: 0.7.0.release, 0.7.1.release)
  Some externally hosted files were ignored as access to them may be unreliable (use --allow-external scoop to allow).
  No matching distribution found for scoop

Python version is 2.7.9

This error is repeated for each node I am trying to distribute work to.

[2019-07-17 14:34:28,536] workerLaunch (127.0.0.1:36053) WARNING Could not successfully launch the remote worker on pn033.
Requested remote group process id, received:
b''
Group id decoding error:
invalid literal for int() with base 10: ''
SSH process stderr:

/.../venv/bin/python: error while loading shared libraries: libpython3.6m.so.1.0: cannot open shared
object file: No such file or directory
/.../venv/bin/python: error while loading shared libraries: lib
python3.6m.so.1.0: cannot open shared object file: No such file or directory
/.../venv/bin/python: err
or while loading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory
/.../venv/bin/python: error while loading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No such file o
r directory

I'm trying to use scoop in a cluster that uses SLURM. I'm trying to run the example you provide in the documentation (helloworld example). I've run the example in the head node with few cpu's and it works (so it seems installation is correct up to some level at least), but when I run it through sbatch it returns the following error:

EXECUTE PYTHON .PY FILE Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/user/.local/lib/python2.7/site-packages/scoop/main.py", line 21, in main() File "/home/user/.local/lib/python2.7/site-packages/scoop/launcher.py", line 454, in main args.external_hostname = [utils.externalHostname(hosts)] File "/home/user/.local/lib/python2.7/site-packages/scoop/utils.py", line 101, in externalHostname hostname = hosts[0][0] IndexError: list index out of range END OF JOBS

In the documentation I read scoop is compatible with slurm, is there a particular configuration step that is not documented (the SSH keys are already configured)?

Thanks,

What version of the product are you using? On what operating system?

$ python -V
Python 2.7.7

$ lsb_release -a
LSB Version:    
:base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd6
4:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 6.4 (Santiago)
Release:        6.4
Codename:       Santiago

What steps will reproduce the problem?
1. Download 0.7.1 tarball, open it and cd into the directory 

2. Verify that pyzmq is available:

python -c "import zmq; print 'It is there'"

3. Try to install scoop with:

python setup.py install --prefix=$MY_INSTALL_DIR

What is the expected output? What do you see instead?

(....a bunch of irrelevant output....)

error: command 'gcc' failed with exit status 1

Failed with default libzmq, trying again with /usr/local
************************************************
Configure: Autodetecting ZMQ settings...
    Custom ZMQ dir:       /usr/local
Assembler messages:
Fatal error: can't create 
build/temp.linux-x86_64-2.7/scratch/tmp/easy_install-L0czGy/pyzmq-14.3.1/temp/ti
mer_create9prql0.o: No such file or directory
build/temp.linux-x86_64-2.7/scratch/vers.c:4:17: fatal error: zmq.h: No such 
file or directory
 #include "zmq.h"
                 ^
compilation terminated.

The problem is that zmq.h is in a non-standard location, available to python 
but not to scoop. 

What is the recommended way of telling scoop where zmq is located?

PS: this is related to issue 8, but that has been closed without anybody being 
assigned to it and just dismissing as a pyzmq installation problem (which is 
not: both pyzmq and zmq are correctly installed and used by other software on 
this system)

Original issue reported on code.google.com by [email protected] on 15 Jul 2014 at 5:04

The http://pyscoop.org/ URL currently redirects to a page saying the domain has expired and is up for renewal. It is currently in the autoRenewPeriod status.

@soravux Thank you for building this amazing library that has made my life as well as many others' lives much easier. I hope you can continue to maintain and contribute to this very useful piece of software. However, if you have decided to abandon this project and no longer intend to renew the domain name, please contact me ASAP. I would be happy to take over the domain name registration and the full-time maintenance of the project if needed. I have a vested interest in the continued survival of this project.

Hi, I have developed my algorithm using DEAP with SCOOP and tested on our little cluster using PBS system. Now, I want to run on our big cluster which installed IBM LSF. Since I cannot specify the accessible hosts, I found SCOOP cannot detect the LSF system. Then, I found the following instruction.

SCOOP natively supports Sun Grid Engine (SGE), Torque (PBS-compatible, Moab, Maui) and SLURM. That means that a minimum launch file is needed while the framework recognizes automatically the nodes assigned to your task.

So, I wonder if there any possibility to support IBM LSF system?

Thank you very much!

I have noted the following problem.

I have code in my .bashrc that echo's a status message. When ssh-ing to this machine while dispatching jobs, SCOOP appears to be confused by this message and appears unable to launch jobs on this host effectively. (I get an error message specifically mentioning the bashrc status message. From what I could see, Only one job per node could be launched in such a case. Is there any way to fix this without having to disable the status message? (The message is rather useful when sshing to the machine in other contexts). Upon disabling the output, the error message disappears and the scheduling works fine.

When $SLURM_JOB_NODELIST is e.g. "nodes[006,011]" I get the following error:

File "/python27/lib/python2.7/site-packages/scoop/utils.py", line 209, in parseSLURM bmin,bmax = rng.split('-') ValueError: need more than 1 value to unpack

If $SLURM_JOB_NODELIST is e.g. "nodes[021-022]" the workers are deployed over the two hosts.

Looks like this project did not receive any updates recently, the web page http://pyscoop.org/ which is in the description and the README redirects to some news site. If the authors are still active on github, could they please comment on the status of the project?

Is there any documentation for how to use scoop with SLURM?

One of the main things I'm wondering about is whether to provide a hosts file to scoop or not when running it from SLURM. Does it automatically figure out the hosts and run simulations on them otherwise?

#!/bin/bash
#SBATCH [email protected]
#SBATCH --mail-type=ALL
#SBATCH --nodes=7
#SBATCH --ntasks=72
#SBATCH --time=99:00:00
#SBATCH --mem=10G
#SBATCH --output=python_job_slurm.out

# Which one is correct?
python -m scoop --hostfile hosts.txt my-script.py
python -m scoop my-script.py

and I run it with sbatch python.slurm

I couldn't use ssh host name.

I checked the parameter to pass the method getaddrinfo.

scoop.BROKER.externalHostname returned thx and scoop.BROKER.task_port returned random int.

How to use other computer through ssh with ~/.ssh/config file? It looks not supported as long as I check the process.

Environment

• Ubuntu 16.04 x64
• Python 2.7.12
• Scoop 0.7.2.0
• ssh thx is properly working.

thx@thx-Prime:~/workspace/scoop$ python -m scoop --host thx -vv scoop_test.py
[2016-12-21 12:11:44,449] launcher  INFO    SCOOP 0.7 2.0 on linux2 using Python 2.7.12 (default, Oct 21 2016, 22:26:43) [GCC 5.4.0 20160609], API: 1013
[2016-12-21 12:11:44,449] launcher  INFO    Deploying 1 worker(s) over 1 host(s).
[2016-12-21 12:11:44,449] launcher  DEBUG   Using hostname/ip: "thx" as external broker reference.
[2016-12-21 12:11:44,449] launcher  DEBUG   The python executable to execute the program with is: /home/thx/.pyenv/versions/2.7.12/bin/python.
[2016-12-21 12:11:44,449] launcher  INFO    Worker d--istribution:
[2016-12-21 12:11:44,449] launcher  INFO       thx:     0 + origin
[2016-12-21 12:11:44,449] brokerLaunch DEBUG   Launching remote broker: ssh -tt -x -oStrictHostKeyChecking=no -oBatchMode=yes -oUserKnownHostsFile=/dev/null -oServerAliveInterval=300 thx /home/thx/.pyenv/versions/2.7.12/bin/python -m scoop.broker.__main__ --echoGroup --echoPorts --backend ZMQ
[2016-12-21 12:11:44,750] brokerLaunch DEBUG   Foreign broker launched on ports 46544, 45126 of host thx.
                                                                                                         [2016-12-21 12:11:44,750] launcher  DEBUG   Initialising remote origin worker 1 [thx].
                                                                                                                                                                                               [2016-12-21 12:11:44,751] launcher  DEBUG   thx: Launching '/home/thx/.pyenv/versions/2.7.12/bin/python -m scoop.launch.__main__ 1 3 --size 1 --workingDirectory "/home/thx/workspace/scoop" --brokerHostname 127.0.0.1 --externalBrokerHostname thx --taskPort 46544 --metaPort 45126 --origin --backend=ZMQ -vvv scoop_test.py'
                                                                                      Warning: Permanently added '192.168.21.10' (ECDSA) to the list of known hosts.
Launching 1 worker(s) using /bin/bash.
Executing '['/home/thx/.pyenv/versions/2.7.12/bin/python', '-m', 'scoop.bootstrap.__main__', '--size', '1', '--workingDirectory', '/home/thx/workspace/scoop', '--brokerHostname', '127.0.0.1', '--externalBrokerHostname', 'thx', '--taskPort', '46544', '--metaPort', '45126', '--origin', '--backend=ZMQ', '-vvv', 'scoop_test.py']'...
Traceback (most recent call last):
  File "/home/thx/.pyenv/versions/2.7.12/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/thx/.pyenv/versions/2.7.12/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/thx/.pyenv/versions/2.7.12/lib/python2.7/site-packages/scoop/bootstrap/__main__.py", line 298, in <module>
    b.main()
  File "/home/thx/.pyenv/versions/2.7.12/lib/python2.7/site-packages/scoop/bootstrap/__main__.py", line 92, in main
    self.run()
  File "/home/thx/.pyenv/versions/2.7.12/lib/python2.7/site-packages/scoop/bootstrap/__main__.py", line 285, in run
    futures_startup()
  File "/home/thx/.pyenv/versions/2.7.12/lib/python2.7/site-packages/scoop/bootstrap/__main__.py", line 266, in futures_startup
    run_name="__main__"
  File "/home/thx/.pyenv/versions/2.7.12/lib/python2.7/site-packages/scoop/futures.py", line 65, in _startup
    result = _controller.switch(rootFuture, *args, **kargs)
  File "/home/thx/.pyenv/versions/2.7.12/lib/python2.7/site-packages/scoop/_control.py", line 199, in runController
    execQueue = FutureQueue()
  File "/home/thx/.pyenv/versions/2.7.12/lib/python2.7/site-packages/scoop/_types.py", line 264, in __init__
    self.socket = Communicator()
  File "/home/thx/.pyenv/versions/2.7.12/lib/python2.7/site-packages/scoop/_comm/scoopzmq.py", line 70, in __init__
    info = socket.getaddrinfo(scoop.BROKER.externalHostname, scoop.BROKER.task_port)[0]
socket.gaierror: [Errno -2] Name or service not known
Exception AttributeError: "'FutureQueue' object has no attribute 'socket'" in <bound method FutureQueue.__del__ of <scoop._types.FutureQueue object at 0x7f5abe703a90>> ignored
Connection to 192.168.21.10 closed.
[2016-12-21 12:11:45,344] launcher  INFO    Root process is done.
                                                                 [2016-12-21 12:11:45,345] workerLaunch DEBUG   Closing workers on thx (1 workers).
                                                                                                                                                   [2016-12-21 12:11:45,345] brokerLaunch DEBUG   Closing broker on host thx.
        Warning: Permanently added '192.168.21.10' (ECDSA) to the list of known hosts.

In Python 3.10 this class has been moved.

from collections import Iterable is now from collections.abc import Iterable

Error message:

Traceback (most recent call last):
  File ".../app/onemax_island_scoop.py", line 28, in <module>
    from scoop import futures
  File ".../venv/lib/python3.10/site-packages/scoop/futures.py", line 19, in <module>
    from collections import namedtuple, Iterable
ImportError: cannot import name 'Iterable' from 'collections' (/opt/homebrew/Cellar/[email protected]/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py)

I used Openssh to connect two win10 computer. The connection works with OpenSSH. but when I use remote computer I get an error: _ Requested remote group process id, received: b'' Group id decoding error: invalid literal for int() with base 10:‘’ SSH process stderr: b'\xb4\xcb\xca\xb1\xb2\xbb\xd3\xa6\xd3\xd0 )\xa1\xa3\r\n' _

my command is : python -m scoop --hostfile hostsUse -vv -n 8 SCOOPTest.py

When I use scoop in centos, no matter how many do I set the number of -n ,My cpu usage is 100%, but cpu load is very normal in macos. ps. My centos machine is 40 cores

how does it happened?

Hello! I run the example in the document, the code is as the following: """test.py""" from math import hypot from random import random from scoop import futures import time def test(tries): return sum(hypot(random(), random()) < 1 for _ in range(tries)) def calcPi(nbFutures, tries): expr = futures.map(test, [tries] * nbFutures) return 4. * sum(expr) / float(nbFutures * tries) if __name__ == "__main__": bt = time.time() print("pi = {}".format(calcPi(3000, 5000))) print('time:', time.time() - bt)

I run this code in the windows cmd with the command "python -m scoop test.py", however, it cost 33 seconds. Then I used "map" instead of "futures.map", it just cost 4 seconds.

What should I do to let SCOOP be able to speed up my code?

Thanks.

Hello! When I used "python -m scoop deap_ga_onemax.py" to run the example file "deap_ga_onemax.py", I found that is much slower than just used "python deap_ga_onemax.py" to run it. Why is that happened? What should I do to use scoop to speed up the code of "deap_ga_onemax.py"?

I am trying to using SCOOP and DEAP for the optimization of a FEA model. In python the model is run by subprocess.run (["./executer"]) . Using python's multiprocess.pool way, I can successfully run the external executor using subprocess.run. But when lunched by SCOOP, the subprocess.run soon skipped without waiting for the modeling result though I can see the external executor was running on the background. There is no error information while running. Please tell me how to solve this? Thank you!