Py4J enables Python programs to dynamically access arbitrary Java objects

@bartdag : Env: win7-x64, Python Version: Python 2.7.12 Py4jVersion :0.10.3 java version "1.8.0_101"

Java Code: LogDebugCmd.java:

public class LogDebugCmd implements Command {
  private Class clazz;
  private String format;
  private Object[] args;

  public LogDebugCmd() {}
  public Command setLog(String format, Object... args) {
    this.format = format;
    this.args = args.clone();
    clazz = getClass();
    return this;
  }

  @Override
  public void execute() {
    VLog.getLog(clazz).debug(format, args);
  }

CommandEntryPoint.java

public class CommandEntryPoint {
  private LogDebugCmd logDebugCmd;

  public CommandEntryPoint() {
    logDebugCmd = new LogDebugCmd();

public LogDebugCmd getLogDebugCmd() {
    return logDebugCmd;
  }
public static void main(String args[]) {
    GatewayServer gatewayServer = new GatewayServer(new CommandEntryPoint());
    gatewayServer.start();
    System.out.println("Gateway Server Started");
  }

Python code test.py:

from py4j.java_gateway import JavaGateway, GatewayParameters
gateway = JavaGateway(gateway_parameters=GatewayParameters(port=25335))
logDebugCmd = gateway.entry_point.getLogDebugCmd()
logDebugCmd.setLog("Hi").execute()

I have my JVM running (through IntelliJ) and then when I run the python code, I am getting the following error : Traceback (most recent call last):

  File "C:/Python27/Lib/site-packages/py4j-0.10.3-py2.7.egg/py4j/sampleTLStest.py", line 3, in <module>
    logDebugCmd = gateway.entry_point.getLogDebugCmd()
  File "C:\Python27\lib\site-packages\py4j-0.10.3-py2.7.egg\py4j\java_gateway.py", line 1131, in __call__
    answer = self.gateway_client.send_command(command)
  File "C:\Python27\lib\site-packages\py4j-0.10.3-py2.7.egg\py4j\java_gateway.py", line 881, in send_command
    connection = self._get_connection()
  File "C:\Python27\lib\site-packages\py4j-0.10.3-py2.7.egg\py4j\java_gateway.py", line 829, in _get_connection
    connection = self._create_connection()
  File "C:\Python27\lib\site-packages\py4j-0.10.3-py2.7.egg\py4j\java_gateway.py", line 835, in _create_connection
    connection.start()
  File "C:\Python27\lib\site-packages\py4j-0.10.3-py2.7.egg\py4j\java_gateway.py", line 970, in start
    raise Py4JNetworkError(msg, e)
Py4JNetworkError: An error occurred while trying to connect to the Java server (127.0.0.1:25335)

I've taken the simple approach of sending Py4JJavaExceptions back to the Java side as references, and everything else as a string. This is a big improvement on just being told that something has gone wrong (and perhaps being able to dig out extra info in the logging).

Disclaimer: I couldn't run all the tests -- they just hang on my machine. Possibly related to some missing cleanup code in the tests -- I can see quite a few places where we might fail to shut down some process and then wait indefinitely for it to terminate. Maybe your CI will be able to run these for me?

My two new tests do pass.

After refactoring some code in an application that uses Py4J (simply splitting the code that interacts with Py4J into a separate file), I noticed exceptions like

Exception TypeError: "'NoneType' object is not callable" in <function <lambda> at 0x10213e5f0> ignored

when my application shut down.

Here's a small test case that reproduces the problem with Python 2.7.1 (on my machine, at least):

In test.py:

from py4j.java_gateway import JavaGateway
gateway = JavaGateway.launch_gateway(die_on_exit=True)

Running python test.py produces no errors. In test2.py, I simply import test:

import test

Running python test2.py results in

Exception TypeError: 'isinstance() arg 2 must be a class, type, or tuple of classes and types' in <function <lambda> at 0x1069f1758> ignored

The cause appears to be related to the order in which modules are cleaned up when Python shuts down. When running test.py, the JavaGateway instance (in __main__) is cleaned up before any of the py4j modules are cleaned up:

$ python -v test.py
...
# cleanup __main__
# cleanup[1] py4j
...
# cleanup[1] py4j.protocol

Compare this to running test2.py, which imports test:

$ python -v test2.py
...
# cleanup __main__
# cleanup[1] py4j
...
# cleanup[1] py4j.finalizer
...
# cleanup[1] py4j.java_collections
...
# cleanup[1] py4j.protocol
...
# cleanup[1] test
...
# cleanup[1] py4j.java_gateway
# cleanup[1] py4j.compat
...

In this case, py4j modules are cleaned up before the test module and the JavaGateway instance are cleaned up.

The cause of this particular TypeError exception isn't immediately clear because of the lambda function. After renaming the lambda functions in Py4J, it appears that it refers to the weakref.ref callback in JavaObject. I suspect that this problem may be due to the weakref callback calling functions from modules that may have already been cleaned up. It looks like this specific 'isinstance() arg 2 must be a class, type, or tuple of classes and types' exception might be occurring in a isinstance(s, unicode) call in smart_decode, which is odd.

Interestingly, adding an extra, unused py4j import in test.py fixes the exception and causes all of the py4j modules to be cleaned up after the test module and JavaGateway instance.

I'm not sure if this is a serious issue:

• It requires a fairly specific pattern of imports and module dependencies to reproduce.
• It only occurs during Python shutdown.
• There's an easy (if non-obvious) workaround.

I was baffled when my PDFs from JasperReports showed up blank. After a lot of head scratching, I found that if I wrote the PDF data to disk on the Java side, it was fine. But if I transferred the data as byte[] over the network link, it would get "wrinkles" on the way, rendering it useless. Wrinkles here meaning that certain bytes (0xD8-0xDF range) would instead be replaced with nulls (0x00). It took me one long sleepless night to track down this problem, but I finally nailed it. It happens on the Java side of things, when the response object is being written to the network socket. The UTF-8 encoder does not like certain characters which happen to be on the reserved range for UTF-16 surrogates which should always come in pairs. This causes the encoder to silently replace some characters with question marks, which in turn either silently corrupts the data or causes an exception on the Python side on decoding. I concluded that this was a protocol design flaw.

More information at: http://en.wikipedia.org/wiki/Mapping_of_Unicode_characters#Surrogates

Py4J JavaGateway (half-)closes CallbackConnection after 30 seconds of inactivity and creates new one next time, but that connection doesn't get closed on the python CallbackServer side and keep piling up until it exhausts all ports.

Reported by Antony Mayi in SPARK-12617

Hi Barthelemy,

With the latest java code, when I forcibly kill the Python client, I'm consistently seeing this CME in the java-side GatewayServer:

WARNING: Error occurred while waiting for a command. java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:886) at java.util.ArrayList$Itr.next(ArrayList.java:836) at py4j.GatewayServer.shutdown(GatewayServer.java:667) at py4j.GatewayServer.shutdown(GatewayServer.java:649) at py4j.commands.ShutdownGatewayServerCommand.execute(ShutdownGatewayServerCommand.java:66) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:744)

Not sure if this warning is expected, but I suspect not as it could leave things in an unclean state upon forcible remote disconnect.

Hi, everybody here. I got a problem when I tried the example code of "py4j".

The code is as follow: from py4j.java_gateway import JavaGateway gateway = JavaGateway() random = gateway.jvm.java.util.Random()

The error information is as follow: Traceback (most recent call last): File "C:\Users\YM_MS\Desktop\py4j_new_test.py", line 3, in random = gateway.jvm.java.util.Random() File "C:\Python27\lib\site-packages\py4j\java_gateway.py", line 708, in getattr '\n' + END_COMMAND_PART) File "C:\Python27\lib\site-packages\py4j\java_gateway.py", line 339, in send_command connection = self._get_connection() File "C:\Python27\lib\site-packages\py4j\java_gateway.py", line 295, in _get_connection connection = self._create_connection() File "C:\Python27\lib\site-packages\py4j\java_gateway.py", line 302, in _create_connection connection.start() File "C:\Python27\lib\site-packages\py4j\java_gateway.py", line 409, in start raise Py4JNetworkError(msg) Py4JNetworkError: An error occurred while trying to connect to the Java server

How to deal with this error?

Thanks!

Hi,

It seems that py4j is not working properly with Java 11. I have no issues with Java 8. Do you have plans to support Java versions higher than 8? Inside build configuration, I cannot see details related to Java higher than 8.

Regards, Piotr

On the java side it's possible to define/add a listener for connection events via:

GatewayServer.addListener(serverListenerImpl)

Having this listener on the GatewayServer is very helpful for supporting the dynamics of the OSGi service registry, as connection can be forcibly closed by breaking the socket or the Python process can crash. See the work I've been doing on supporting the dynamics of the OSGi service registry across Python <-> Java here: https://github.com/ECF/Py4j-RemoteServicesProvider

On the Python side it would be useful to have a similar listener...perhaps on CallbackServer.addListener, or via CallbackServerParameters or both?

I'm of the opinion that it should 'look' as much like the GatewayEventListener as possible...for convenience. See some discussion on this issue here:

https://groups.google.com/a/py4j.org/forum/?utm_medium=email&utm_source=footer#!topic/py4j/sBcEfxSrYwk

I would be willing to work/contribute to implementing this if desired. I need some personal resources to do so in a timely manner, however.

This allows a single JVM or Python to have multiple ClientServer objects instantiated and used from the same thread.

I decided to use a Map/dict of integer (incremented per PythonClient/JavaClient) to connection objects. Three reasons of differing validity:

• avoided changing thread local to non-static scope
• no additional references to PythonClient/JavaClient in "hard to see" thread locals
• PythonClient/JavaClient were not obviously guaranteed to stay hashable

There is a downcast in ClientServerConnection.setThreadConnection that could use a review.

The thread of Python callback server is started without 'daemon=True', then then process can not exit without shutdown the callback server explicitly.

Suppose you have a function within Java which returns an immutable list. As simple as

class Test {
    public static List<Integer> anImmutableList() {
        return List.of(1);
    }
}

And you attempt to get that list, plus later on iterate through it:

test_list = java_gateway.Test.anImmutableList()
result = [ it + 1 for it in test_list]

Then a Py4JError is returned whenever it is attempted to iterate through the list:

py4j.protocol.Py4JError: An error occurred while calling o1093.iterator. Trace:
java.lang.reflect.InaccessibleObjectException: Unable to make public java.util.Iterator java.util.ImmutableCollections$AbstractImmutableList.iterator() accessible: module java.base does not "opens java.util" to unnamed module @417bda74
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:199)
	at java.base/java.lang.reflect.Method.setAccessible(Method.java:193)
	at py4j.reflection.MethodInvoker$1.run(MethodInvoker.java:240)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:238)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.base/java.lang.Thread.run(Thread.java:833)

At the time of writing Java 17 and py4j 0.10.9.7 is used.

Is this an actual bug or am I missing something? Somehow this works very counter intuitive (one needs to create a mutable list forcefully, which then introduces code style and quality issues)

in py4j-java/src/main/java/py4j/JavaServer.java there must use the defaultaddress() to set the java address, when in docker, this may cause problem of connection, so need to add constructor that can pass the java address to JavaClient

This PR addresses issue #374. It provides a way for users to specify how they would like Python integers to be converted into Java (whether to treat them like an int, short, or long). This PR currently only adds an integer and long variant, since Py4J does not have general support for shorts.

This PR fixes the logger used in exception handling when sending commands. Previously the root logger was used, and with this fix the py4j specific logger is used.

This PR makes py4j give more explicit names to the threads it creates (like py4j-… rather than the default Thread-…), which makes it easier to inspect JVM processes where py4j is running.

It also marks those threads as daemon, so that exiting the main thread of the JVM, by reaching the end of the main method, actually makes the JVM exit (provided no other non-daemon threads were created by users).