This pull request removes translate.py and all version-specific translation.

With the last pull request, I thought that the translation layer would still be useful, but I now think it's more of a hassle than it's worth. In Cassandra 0.7, you have to set the replication_factor attribute of a KsDef. In 1.1, the request will fail if you set it. This means that at the least, we would need to track those versions and change behavior in Telephus based on the api_version. Furthermore, replication_factor isn't even needed with NetworkTopologyStrategy, so it just adds a possibility of cluttering things there.

The translation later might have been useful for the 0.7 to 0.8 transition, but I think it's time for it to go. It provides very little benefit at this point and has the downside of being potentially confusing.

Telephus is hands-off enough in its use of the Cassandra api that it ought to be able to cleanly support all use of the api with any version of Cassandra past 0.7. That is, a user should be able to use the latest Telephus with Cassandra 0.7, assuming the user is aware of what methods exist on the server side (I'm not suggesting Telephus simply abstract all the differences between the Cassandra versions, just that they should all be usable).

Right now, it's close to that goal, but falls short in a few instances:

• it thinks that only 0.7 and 0.8 exist. this causes some difficulties with the existing translate.py code (which, I guess, is just there to make KsDef.replication_factor appear to keep working?)
• tests assume that only 0.8 supports counters, although anything after 0.7 does
• the thrift defs for 1.0 and 1.1 aren't present
• user can't select the api version to use before connecting

In an ideal world, it would be sufficient to use the 1.1-beta1 thrift defs for all versions, since it's supposed to be backwards compatible. However, quite a few CfDef and KsDef attributes have been removed since 0.7, so a user with only the 1.1-beta1 thrift code wouldn't be able to set those attributes for an 0.7 Cassandra.

If we don't care about setting those options with Telephus, then we can get rid of the multiple-thrift-def thing altogether. That's probably not an option for at least some telephus users I can think of, though.

This pull request also includes the changes from the pull request: https://github.com/driftx/Telephus/pull/31

That request should be merged in first before reviewing this one.

The changes here should be pretty simple. Versions of cassandra > 0.7 include an rpc_address field in the output of describe_ring. When using describe ring for purposes of connection pooling/failover the rpc address should be used when attempting to connect to auto discovered nodes, assuming it is a good ip and not '0.0.0.0'.

There is an included unit test that should cover all added functionality. Additionally I've manually tested against both 0.7, and 0.8 where nodes had differing rpc and listen addresses.

Try using Иона as the column name in the tests and you'll see the following exception:

Traceback (most recent call last):
  File "/Users/mwilliamson/Projects/Telephus/telephus/protocol.py", line 200, in reqError
    raise err.value
exceptions.UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)

don't forget to add the encoding declaration in the test file:

# -*- coding: utf-8 -*-

See: https://issues.apache.org/jira/browse/CASSANDRA-3187

I also update the unit tests to work correctly with 1.0. Ran the unit tests against 0.7/0.8/1.0.

When using telephus from code that uses defer.inlineCallbacks, all exceptions show a traceback that is internal to telephus, and they never show the line in my code which generated the error.

Example:

from twisted.internet import defer, reactor
from telephus.protocol import ManagedCassandraClientFactory
from telephus.client import CassandraClient
from telephus.cassandra.ttypes import KsDef, CfDef, InvalidRequestException

KEYSPACE = "test_keyspace"
CF_NAME = "cf"

def raise_error(_):
    raise Exception("i failed")

@defer.inlineCallbacks
def error():
    d = defer.Deferred()
    reactor.callLater(1, d.callback, True)
    d.addCallback(raise_error)
    yield d

@defer.inlineCallbacks
def run():
    cmanager = ManagedCassandraClientFactory()
    client = CassandraClient(cmanager, KEYSPACE)
    reactor.connectTCP("localhost", 54070, cmanager)
    yield cmanager.deferred
    ksdef = KsDef(
                name=KEYSPACE,
                strategy_class='org.apache.cassandra.locator.SimpleStrategy',
                replication_factor=1,
                cf_defs=[ CfDef(
                    keyspace=KEYSPACE,
                    name=CF_NAME,
                    column_type="Standard")
                        ])
    #yield error()
    yield client.system_add_keyspace(ksdef)

if __name__ == "__main__":
    reactor.callWhenRunning(run)
    reactor.run()

In this example, the output, when run for a second time, will be: Unhandled error in Deferred: Traceback (most recent call last): Failure: telephus.cassandra.ttypes.InvalidRequestException: InvalidRequestException(why='Keyspace already exists.')

I would expect to see the line yield client.system_add_keyspace(ksdef) somewhere in the backtrace, but it is not there.

But if we uncomment the #yield error() line, backtraces show as expected.

This issue is specially bad for actions like insert, where you can get a traceback like this:

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 1252, in put
    self.waiting.pop(0).callback(obj)
  File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 318, in callback
    self._startRunCallbacks(result)
  File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 441, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
--- <exception caught here> ---
  File "/usr/local/lib/python2.6/dist-packages/telephus/protocol.py", line 210, in _process
    d = proto.submitRequest(request)
  File "/usr/local/lib/python2.6/dist-packages/telephus/protocol.py", line 92, in submitRequest
    d = fun(*request.args)
  File "/usr/local/lib/python2.6/dist-packages/telephus/cassandra/Cassandra.py", line 730, in insert
    self.send_insert(key, column_parent, column, consistency_level)
  File "/usr/local/lib/python2.6/dist-packages/telephus/cassandra/Cassandra.py", line 741, in send_insert
    args.write(oprot)
  File "/usr/local/lib/python2.6/dist-packages/telephus/cassandra/Cassandra.py", line 4044, in write
    oprot.trans.write(fastbinary.encode_binary(self, (self.__class__, self.thrift_spec)))
exceptions.TypeError: expected string or Unicode object, int found

and never know which part of your code failed.

Hi folks,

I've tried using the release python-telephus_0.7~beta3.3.tar.gz from http://debian.riptano.com/debian/pool/. I installed telephus into a new virtualenv and I used easy_install to install thrift. The thrift version is 0.2.0. I'm also running cassandra-0.7 on the host.

I run the example.py file and I get the following stack trace below. Does anybody have an idea where this error is coming from? I think the error might be coming from the fastbinary.encode_binary call? Can anybody else reproduce this?

Thanks, Matt Rodriguez

2010-12-03 15:40:23-0800 [-] Starting factory <telephus.protocol.ManagedCassandraClientFactory instance at 0x102900290> 2010-12-03 15:40:23-0800 [Uninitialized] Unhandled error in Deferred: 2010-12-03 15:40:23-0800 [Uninitialized] Unhandled Error Traceback (most recent call last): File "/Users/mateo/src/Telephus/telephus/protocol.py", line 219, in submitRequest return self.queue.get().addCallback(_process) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py", line 195, in addCallback callbackKeywords=kw) File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py", line 186, in addCallbacks self._runCallbacks() File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, _args, *_kw) --- --- File "/Users/mateo/src/Telephus/telephus/protocol.py", line 210, in _process d = proto.submitRequest(request) File "/Users/mateo/src/Telephus/telephus/protocol.py", line 92, in submitRequest d = fun(*request.args) File "/Users/mateo/src/Telephus/telephus/cassandra/Cassandra.py", line 730, in insert self.send_insert(key, column_parent, column, consistency_level) File "/Users/mateo/src/Telephus/telephus/cassandra/Cassandra.py", line 741, in send_insert args.write(oprot) File "/Users/mateo/src/Telephus/telephus/cassandra/Cassandra.py", line 4044, in write oprot.trans.write(fastbinary.encode_binary(self, (self.class, self.thrift_spec))) exceptions.TypeError: an integer is required

Good afternoon,

Cassandra 0.7 has a feature to create and remove keyspaces at runtime. I have an application that uses Telephus as a client to our Cassandra datastore. We would like to create and remove keyspaces.

I've noticed that the unstable branch has this ability to create and remove keyspaces (system_add_keyspace, system_drop_keyspaces). I see there are unittests, but I haven't tried to run them yet. Are these methods working? When do you think these changes will be merged into the master branch? I might be able to provide some of my time to merge these changes into the master branch.

Thanks, Matt

Hello, I am using your test/test_cassandraclient.py as a reference, but all the values are like "test" and "test2". Could you please update these to something that describes what they are, e.g. "value test" or something so we can tell what the arguments are and that we aren't retrieving a wrong column with the same value.

Hi! I use the following function to create a new user:

def createUser(self, identity, email, fname, lname, mname):
    cols = [
        Column(name="email", value=email, timestamp=datetime.datetime.now()),
        Column(name="fname", value=fname, timestamp=datetime.datetime.now()),
        Column(name="lname", value=lname, timestamp=datetime.datetime.now()),
        Column(name="mname", value=mname, timestamp=datetime.datetime.now())
    ]
    return self.db.batch_mutate(mutationmap={identity: {"users": cols}})

But after calling it I have an error. Stack trace:

Traceback (most recent call last): File "server.py", line 100, in handle_request res = yield self.backend.createUser(identity, email, fname, lname, mname) File "/usr/local/lib/python2.7/site-packages/Twisted-11.1.0-py2.7-freebsd-8.2-STABLE-amd64.egg/twisted/internet/defer.py", line 545, in _runCallbacks current.result = callback(current.result, _args, *_kw) File "build/bdist.freebsd-8.2-STABLE-amd64/egg/telephus/pool.py", line 301, in d.addCallback(lambda _: method(*req.args)) File "build/bdist.freebsd-8.2-STABLE-amd64/egg/telephus/cassandra/Cassandra.py", line 823, in batch_mutate self.send_batch_mutate(mutation_map, consistency_level) File "build/bdist.freebsd-8.2-STABLE-amd64/egg/telephus/cassandra/Cassandra.py", line 832, in send_batch_mutate args.write(oprot) File "build/bdist.freebsd-8.2-STABLE-amd64/egg/telephus/cassandra/Cassandra.py", line 4423, in write oprot.trans.write(fastbinary.encode_binary(self, (self.class, self.thrift_spec))) TypeError: an integer is required

What can cause the problem?

test.test_cassandraclient
  CassandraClientTest
    test_bad_params ...                                               [SKIPPED]
    test_batch_insert_get_slice_and_count ...                              [OK]
    test_batch_mutate_and_remove ...                                       [OK]
    test_batch_mutate_with_deletion ...                                    [OK]
    test_column_family_manipulation ...                                  [FAIL]
    test_counter_add ...                                              [SKIPPED]
    test_counter_remove ...                                           [SKIPPED]
    test_describes ...                                                     [OK]
    test_errback ...                                                       [OK]
    test_indexed_slices ...                                                [OK]
    test_insert_get ...                                                    [OK]
    test_keyspace_manipulation ...                                         [OK]
    test_multiget_slice_remove ...                                         [OK]
    test_range_slices ...                                                  [OK]
    test_ttls ...                                                          [OK]
  ManagedCassandraClientFactoryTest
    test_api_match ...                                                     [OK]
    test_initial_connection_failure ...                                    [OK]

===============================================================================
[SKIPPED]
Disabled pending further investigation...

test.test_cassandraclient.CassandraClientTest.test_bad_params
===============================================================================
[SKIPPED]
Counters are not supported in 0.7

test.test_cassandraclient.CassandraClientTest.test_counter_add
test.test_cassandraclient.CassandraClientTest.test_counter_remove
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1039, in _inlineCallbacks
    result = g.send(result)
  File "/var/lib/jenkins/jobs/Telephus/workspace/test/test_cassandraclient.py", line 366, in test_column_family_manipulation
    self.assertEqual(cfdef, cfdef2)
  File "/usr/local/lib/python2.7/site-packages/twisted/trial/unittest.py", line 270, in assertEqual
    % (msg, pformat(first), pformat(second)))
twisted.trial.unittest.FailTest: not equal:
a = CfDef(comment='foo', key_validation_class=None, min_compaction_threshold=5, key_cache_save_period_in_seconds=3600, gc_grace_seconds=86400, default_validation_class='org.apache.cassandra.db.marshal.BytesType', max_compaction_threshold=31, read_repair_chance=1.0, key_cache_size=200000.0, id=1016, replicate_on_write=None, subcomparator_type=None, merge_shards_chance=None, row_cache_provider=None, row_cache_save_period_in_seconds=0, column_type='Standard', memtable_throughput_in_mb=None, memtable_flush_after_mins=None, column_metadata=[], key_alias=None, name='TransientCF', keyspace='TelephusTests', memtable_operations_in_millions=None, comparator_type='org.apache.cassandra.db.marshal.BytesType', row_cache_size=0.0)
b = CfDef(comment='foo', key_validation_class=None, min_compaction_threshold=5, key_cache_save_period_in_seconds=None, gc_grace_seconds=86400, default_validation_class='org.apache.cassandra.db.marshal.BytesType', max_compaction_threshold=31, read_repair_chance=1.0, key_cache_size=200000, id=1016, replicate_on_write=None, subcomparator_type=None, merge_shards_chance=None, row_cache_provider=None, row_cache_save_period_in_seconds=None, column_type='Standard', memtable_throughput_in_mb=None, memtable_flush_after_mins=None, column_metadata=[], key_alias=None, name='TransientCF', keyspace='TelephusTests', memtable_operations_in_millions=None, comparator_type='org.apache.cassandra.db.marshal.BytesType', row_cache_size=0)


test.test_cassandraclient.CassandraClientTest.test_column_family_manipulation
-------------------------------------------------------------------------------
Ran 17 tests in 12.956s

FAILED (skips=3, failures=1, successes=13)```

There are small typos in:

• telephus/cassandra/Cassandra.py

Fixes:

• Should read independently rather than idependently.
• Should read immediately rather than immediatily.

Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md