This branch is to fix a problem discussed starting here in Campfire, and explained further in bug 7101.

To solve this, I replaced the current locking solution with flock, as in the other libexec scripts.

I'm fairly confident about this part of the patch, but I noticed there's some locking in dhcp-conf-alter.py. I don't think there should be any side effects, but I don't know. Hence, the pull request.

A Label on the blankspace partition will allow mounting via Label instead of device, this makes it easier in fstab where the device of the blankspace partition may change.

When terminating an instance, any global status query and any status query on the instance hangs.

This is caused by the WorkspaceHomeImpl:destroy method taking a per-instance lock for the duration of the whole termination. This includes a lengthy call to the workspace control agent. The WorkspaceHomeImpl:find method also tries to take this lock and hangs while the termination is in progress.

We have noticed an issue with lantorrent that causes it to hang and become unresponsive. The situation that causes this can be reproduced by repeating the following steps.

-Start up an instance/cluster either using the cloud-client or ec2 api. -Issue a terminate before the propagation has completed.

After this occurs the database is left with the entries as stale records, which never seem to get updated. If this occurs either after a large termination or after repeated executions of the above process, lantorrent stops copying any new requests. Looking at the lantorrent daemon log shows this error:

2011-11-21 11:37:27,422 - WARNING - Stack trace 2011-11-21 11:37:27,423 - WARNING - =========== 2011-11-21 11:37:27,423 - WARNING - Traceback (most recent call last): File "build/bdist.linux-x86_64/egg/pylantorrent/ltConnection.py", line 114, in send self._write_to_socket(data) File "build/bdist.linux-x86_64/egg/pylantorrent/ltConnection.py", line 99, in _write_to_socket self.socket.sendall(data) File "", line 1, in sendall error: [Errno 104] Connection reset by peer

2011-11-21 11:37:27,423 - WARNING - =========== 2011-11-21 11:37:27,423 - WARNING - <class 'socket.error'>

If we clear out the req.db file and restart lantorrent, all seems to be well again. This is occurring in our production environment of 2.7, as well as devel which is at 2.8.

Please let me know what other information you need in order to help troubleshoot this.

Thanks, Brian

The nimbus-new-user does accept email addresses with the '+' character in the leading portion of the username, which are perfectly legitimate email addresses. For instance [email protected].

When attempt to run nimbus-new-user with such an address, the following error is triggered:

./nimbus-new-user --dest /tmp --batch --web --group=3 --report=cert,key,dn,canonical_id,access_id,access_secret,url,web_id,cloud_properties [email protected] Traceback (most recent call last): File "/opt/nimbus/libexec/nimbus_new_user.py", line 420, in rc = main() File "/opt/nimbus/libexec/nimbus_new_user.py", line 407, in main create_user(o, db) File "/opt/nimbus/libexec/nimbus_new_user.py", line 340, in create_user raise ex1 nimbusweb.setup.setuperrors.UnexpectedError: Problem creating certificate. [<<< stderr: 'Problem creating certificate: String index out of range: -1

With the new version of LANTorrent "repackaged to be more pythonic", it is not possible to do an installation without Internet anymore. The installer tries to fetch Python libraries, like simplejson 2.1.

We need some mechanism to embed the libraries, like it was done with cumulus.

Kyle Fransham suggested enabling virtio for KVM instances in this workspace-user post: http://lists.globus.org/pipermail/workspace-user/2011-March/001630.html

This should be integrated. And more generally, can we make libvirt XML more of a template that can be edited by admins to suit their needs?

Added feature to specify a separate location for the blankspace partition file in the control/etc/workspace-control/images.conf "blankspacedir" left the default value in conf to be secureimages Feature requested from user due to disk performance/space issues on cluster where large amounts of blankspace were needed and the local filesystem only had space for the image files, and using a networked filesystem for both blankspace and instance caused poor IO on the network filesystem.

When using the cloud client, I recently ran into a problem where the 2 node cluster I requested was not provided due to some error. However, when I checked $? immediately after cloud-client returned, it was still 0. In case of failure, cloud-client should return a non-zero exit code to signal failure to any wrapping entity.

Nimbus service goes into infinite loop when requesting a VM that has needs more memory than available on any node... but only for some host/guest memory values.

I can repeat the bug with these values: nodes have 3072 MB of memory, and the VM is requested with 3584 MB. This happens both with the cloud client and the EC2 API.

Excerpt from the logs:

2011-01-29 23:04:38,610 DEBUG defaults.DefaultSlotManagement [ServiceThread-26,reserveSpace:402] request for 1 space(s) with mem = 3584 2011-01-29 23:04:38,613 ERROR defaults.ResourcepoolUtil [ServiceThread-26,getAvailableEntries:267] No resource is available for this request (based on memory). 2011-01-29 23:04:38,614 INFO async.AsyncRequestManagerImpl [ServiceThread-26,releaseSpace:325] 3584MB RAM have to be freed to give space to higher priority requests

(repeated many times)

interlaced with this:

2011-01-29 23:04:38,676 WARN async.AsyncRequestManagerImpl [ServiceThread-26,releaseSpace:331] [NIMBUS-EVENT]: Asynchronous requests are consuming 0MB RAM , but AsyncRequestManager was requested to free 3584MB RAM. Freeing 0MB RAM.

This is disabled right here: https://github.com/nimbusproject/nimbus/blob/master/control/src/python/workspacecontrol/defaults/imageprocurement/propagate_https.py#L185

I don't see any reason we shouldn't allow this. Does anyone have any objections?

With the latest nimbus version the cumulus-deps.tar.gz archive is still downloaded which brings dependencies like pyOpenssl-0.10 and pyOpenSSL-0.13 into the mix. The newest pyOpenSSL version which still compiles against newer openssl versions is 0.14. To be able to compile against a not out of date version of openssl this dependency needs to be updated. Is there any code that requires this old pyOpenSSL versions?

The same issue doesn't repro on boto 2.2.2 and older, it starting appearing on boto 2.32. You can validate by running the rollowing snippet:

import boto
from boto.s3.connection import S3Connection
from boto.s3.connection import OrdinaryCallingFormat
from boto.s3.key import Key

bucket="test"
access_key="ACCESS_KEY"
secret_key="SECRET_KEY"
host="localhost"
port=4288

s3conn = S3Connection(access_key,secret_key, is_secure=False,port=port,host=host, calling_format=OrdinaryCallingFormat())
s3conn.create_bucket(bucket)

# Throws
b = s3conn.get_bucket(bucket)
print(b)

The nimbus log for the issue is:

2014-12-03 07:46:59,616 - 18017 - INFO - Access granted to ID=u87xc9prmanQD8DlvkPWn requestId=8fff49c47ac011e4a339005056b5d1fb uri=/test/
2014-12-03 07:46:59,616 - 18017 - INFO - path /test/
2014-12-03 07:46:59,616 - 18017 - INFO - path /test/ bucket files object None
2014-12-03 07:46:59,623 - 18017 - ERROR - <?xml version="1.0" ?><Error><Code>InvalidArgument</Code><Message>Invalid Argument</Message><Resource>/test/</Resource><RequestId>8fff49c47ac011e4a339005056b5d1fb</RequestId></Error>
2014-12-03 07:46:59,623 - 18017 - ERROR - Stack trace
2014-12-03 07:46:59,624 - 18017 - ERROR - ===========
2014-12-03 07:46:59,624 - 18017 - ERROR - Traceback (most recent call last):
  File "/opt/cumulus/lib/python2.7/site-packages/pycb-0.1-py2.7.egg/pycb/cumulus.py", line 172, in process_event
    self.allowed_event(request, user, requestId, rPath)
  File "/opt/cumulus/lib/python2.7/site-packages/pycb-0.1-py2.7.egg/pycb/cumulus.py", line 184, in allowed_event
    cbR = self.request_object_factory(request, user, path, requestId)
  File "/opt/cumulus/lib/python2.7/site-packages/pycb-0.1-py2.7.egg/pycb/cumulus.py", line 156, in request_object_factory
    raise cbException('InvalidArgument')
cbException: Invalid Argument

2014-12-03 07:46:59,624 - 18017 - ERROR - ===========
2014-12-03 07:46:59,624 - 18017 - ERROR - <class 'pycb.cbException.cbException'>

Hi, I am getting this error after cloud-client.sh --status:

Problem: General error: ; nested exception is: java.net.ConnectException: Connection refused

Since this pull request was merged in boto, calling get_key_pair on Nimbus with an unknown key raises an IndexError.

This is because Nimbus returns an empty array, while boto expects a 400 Bad Request response like Amazon EC2 does.

nimbus-reset-state -e is supposed to reset IaaS databases, the runtime tracking of VM/VMM state, network leases, etc.

It deletes elastic.db, which also contains SSH keypairs uploaded with the EC2 API. Generally an administrator will want to keep these keypairs, so nimbus-reset-state should instead only purge the instances and reservations tables.