I've updated the travis config in 1fa7a9552cf3ff187af5bd2eb137f0d07c77475d and this somewhat improved the situation:

• When built by gcc, only stat_test failed. Seems like a problem of floating-point precision?
• When built by clang, all the tests succeeded.

https://github.com/jubatus/jubatus/blob/develop/src/common/mprpc/rpc_server.cpp#L41

Currently, all MessagePack-RPC clients (C++, Python, Ruby, Java) does not interpret msgpack::rpc::ARGUMENT_ERROR (it is actually a int value "2"). This causes meaningless "RPC Error 2" and it is too unfriendly for users.

I think it's better to send back a string message like "Type mismatch error in argument" instead.

Current ML tasks(classifier, recommender..) implemented in *_serv.cpp that tight coupled with server module.

related #250, I commented refactoring step 1 and 2.

Currently, save method just dumps mixable data structures using pficommon serializer. Points of view that need to be considered may include interoperability with other software and upper compatibility (for example, including the version number).

See also #222.

I imploved get_status method.

Interface

• Not Changed

Changed

• Don't return following keys in standalone mode
  • interval_sec
  • interval_count
  • zk
  • use_cht (0 or 1)
• Add following keys
  • Key: type, Value: machine laerning type (e.g. classifier, anomaly ...)
  • Key: loglevel, Value: Command-line argument or default value (Converted to string like INFO, FATAL ...)
  • Key: logdir, Value: Command-line argument or ""(empty string) (It's default value)
  • Key: configpath, Value:
    • Standalone Mode: Command-line argument
    • Distributed Mode: Node path in zookeeper
  • Key: clock_time, Value: System clock time (epoch time) when receiving a request (sec)
  • Key: uptime, Value: How long the process has been running (sec)
  • Key: start_time, Value: System clock time (epoch time) when process started (sec)
  • Key: pid, Value: Process ID
  • Key: user, Value: User name who started process
  • Key: last_saved, Value: System clock time (epoch time) when saved model file at last
  • Key: last_saved_path, Value: Model file path which was saved at last
  • Key: last_loaded, Value: System clock time (epoch time) when loaded model file at last
  • Key: last_saved_path, Value: Model file path which was loaded at last
  • Followings are returned only in distributed mode
    • Key: name, Value: Command-line argument
    • Key: zookeeper_timeout, Value: Command-line argument or default value
    • Key: interconnect_timeout, Value: Command-line argument or default value
    • Key: connected_zookeeper, Value: Host and Port of zookeeper that process connected (e.g. 127.0.0.1_2181)
    • Key: mixer, Value: Command-line argument or default value
    • Status of mixer

example

In python, output of pprint is followings:

• https://gist.github.com/rimms/8434705

TBD (I will create new issues)

• I want to change values of xxxpath to full-path.
• We should reconsider about status of storages.

Updated 2014/01/20 10:34: User ID -> User name Updated 2014/01/27 15:56: Remove join, Use empty string as default value of logdir, fixed typo (valu -> value)

Current save/load operations support the model reusability only on the same environment where the models are built. On the other hand, users might want to use the models on another environment. Since the saved models are lack of some information to reconstruct the model, especially about what only zookeeper knows but servers doesn't.

For that purpose, we may make save/load more self-contained and compatible for running on the other environments, or, create new interface such as import/export to support such functionalities.

Finally removes google-glog dependency! (fix #746)

~~Note that this pull-req depends on #807 and cannot be merged before #807.~~

• Modifies the following command line options:
  • Adds --log-config option. Users can now specify XML log4cxx configuration file, which can be used for server/proxy/jubavisor.
  • Removes --loglevel option. Users should specify desired log levels in XML configuration file.
  • Modifies --logdir option. This option is now only used for printing ZooKeeper logs.
  • Deprecates --debug option for interactive commands (jubaconfig and jubactl) --debug option used to have a effect of printing logs to standard output instead of files. As I think those commands should not write logs, these commands now always prints their logs to standard output.
• I added #include <errno.h> for some files. This is because errno is being used without including errno.h. As glog/logging.h indirectly includes errno.h, the problem did not become apparent until now.
• Installs example XML configuration files to ${PREFIX}/jubatus/share/example/log/log4cxx.xml.

Users only define return values in the current IDL. When I want to return error codes or something like that, they need to make a user-defined message, which includes a success/fail flag and an error code. It's not cool. I want to define possible error types in IDL, and generated servers and clients must transfer these error objects through RPC.

For examle:

void update_node(0: string id, 1: map<string, string> property) throws unknown_id, invalid_property

This API may raise unknwon_id exception or invalid_property exception.

In server side, we need to catch these user defined exceptions. In client side, check error object and deserialize them to user defined exceptions.

This modification is not difficult but compatibility is destructed.

I tried to measure memory size on before and after call clear_row.

The result is following. The value is RSS size which got by using get_status.

| method | after start-up | after update_row 100K ID | after clear_row 10K ID | | --- | --- | --- | --- | | lsh | 4312 | 521888 | 522048 | | minhash | 4308 | 216708 | 216840 | | euclid_lsh | 4336 | 236324 | 236672 | | inverted_index | 4292 | 434044 | 434192 |

We cannot use jubarecommender long time without some solution (e.g. stop service and add memory ...).

Client socket created and no RPC called for some seconds, the jubatus server is to close the socket. Then the client cannot call RPC because the socket was closed, so timeout_error returned. It is not useful for client, closed socket should try to reconnect when the socket has been closed by server.

Yesterday, the client library always raises request time out error, and the server returns no response. The situation is below:

• The server process is alive
• Even get_stauts method does not respond
• No other clients use the server
• CPU usage of the server is very low
• I forgot put logs to a file ;-(

I suspect that the server gets write-lock, and accidentally it cannot release the lock.

If anyone else face the same situation, please teach me.

dependent on:

• https://github.com/jubatus/jubatus-mpio/pull/21
• https://github.com/jubatus/jubatus-msgpack-rpc/pull/23
• https://github.com/jubatus/jubatus_core/pull/254

related issue: #945

In some cases, Jubatus java client is used in serialized such as storm topology. Current jubatus client throws NoSerializableException through this process. This patch enables jenerator to create Serializable clients and user defined data.