Here is the setup.

Windows 7 x64 sp1 Stackless Python 2.7.* x32 Cython 0.28.3

I am compiling python-openzwave which uses Cython. I get this error when compiling on stackless 2.7.* and not on standard python 2.7.*.

I get a Python app crash as soon as it starts cythonizing.

Here is the data from the app crash window

Problem Event Name:	APPCRASH
Application Name:	python.exe
Application Version:	0.0.0.0
Application Timestamp:	59ccecdf
Fault Module Name:	python27.dll
Fault Module Version:	2.7.14150.1013
Fault Module Timestamp:	59ccecce
Exception Code:	c0000005
Exception Offset:	00069c2a
OS Version:	6.1.7601.2.1.0.256.1
Locale ID:	1033
Additional Information 1:	0a9e
Additional Information 2:	0a9e372d3b4ad19135b953a78882e789
Additional Information 3:	0a9e
Additional Information 4:	0a9e372d3b4ad19135b953a78882e789

The project I am working on has been around since 1996 and Stackless Python was used and is not something that can be removed easily I am also not able to upgrade the python version to 3+ any time soon. I have brought up this error to the author(s) of Cython. they were kind enough to add Stackless to their test build on travis. they found that 2.7 fails completely and 3.5 (i believe) builds successfully. they are not compiling the same code as me so that is now removed from the equation. they are building on linux. so the OS has now been removed. If I take the dll (pyd) that is built on standard 2.7 and bring it over to stackless 2.7 I get the exact same app crash on import. This leads me to believe the problem is not with Cython but instead it is with Stackless Python.

I searched the internet to see if anyone has had an issue with the 2 working together and I could not find any issues. I couldn't find anything on someone running the 2 together either. Pretty much a dead end.

The issue reported to Cython can be viewed here:

https://github.com/cython/cython/issues/2476

Any help is appreciated. Thanks again. Kevin

The Stackman project (http://github.com/kristjanvalur/stackman) aims to provide simple stack switching code for modern platforms. The aim is to have application-agnostic code that performs the basic functionality of saving machine registers and switching stack pointer, leaving the custom code of saving/restoring stack and choosing stack pointer to a user callback.

The resulting switching code is very succinct and easy to verfy.

This pull requests adds Stackman based switching for the supported platforms, which is GCC on intel (x86 and x86_64) and ARM 32 hard float and aarch64.

Originally reported by: Kristján Valur Jónsson (Bitbucket: krisvale, GitHub: kristjanvalur)

Now that python has moved to github, how do we merge stuff

(couldn't figure out how to start a simple chat, without opening an issue)

• Bitbucket: https://bitbucket.org/stackless-dev/stackless/issue/118

Originally reported by: Anselm Kruis (Bitbucket: akruis, GitHub: akruis)

Either my understanding of Python threading is poor or slp_get_frame() is buggy. This is the current implementation.

#!python
PyFrameObject *
slp_get_frame(PyTaskletObject *task)
{
    PyThreadState *ts = PyThreadState_GET();

    return ts->st.current == task ? ts->frame : task->f.frame;
}

It uses the current thread state, not the thread state of the given tasklet. task->cstate->tstate. If this is a bug, it affects tasklet.alive and other properties.

• Bitbucket: https://bitbucket.org/stackless-dev/stackless/issue/44

Originally reported by: RMTEW FULL NAME (Bitbucket: rmtew, GitHub: rmtew)

(originally reported in Trac by @akruis on 2013-11-28 10:46:33)

Thanks to Kristján Stackless Python got a new method: tasklet.bind_thread(). I won't discuss the motivation and the implementation details of this method here. It's all in this thread of the Stackless mailing list: http://www.stackless.com/pipermail/stackless/2013-November/005869.html

The purpose of this ticket is to add all the missing details we need for the next release (2.7.6) of Stackless.

• document the method in Doc/library/stackless/tasklets.rst
• update the documentation in Doc/library/stackless/threads.rst to clearly describe the relation between a tasklet and a thread:
  • A tasklet always has an associated thread. This thread is identified by the property thread_id
  • The thread_id of a tasklet changes only if the method bind_thread() is called or if the associated thread terminates. In the later case the new thread is ... \ @Kristján here I need your input: I noticed, that the thread id changes to the main thread, but what happens if the main thread terminates while other threads are still active?
• update Stackless/changelog.txt

I'll take care about these points myself, but I need help from Kristján to add the missing information.

• Bitbucket: https://bitbucket.org/stackless-dev/stackless/issue/26

Originally reported by: Anselm Kruis (Bitbucket: akruis, GitHub: akruis)

I added two attributes to class tasklet: tasklet.trace_function and tasklet.profile_function. These attributes are the tasklet counterparts of the standard functions sys.gettrace(), sys.settrace(), sys.getprofile() and sys.setprofile(). With these attributes it is now possible to control tracing completely using the schedule callback. An example is given in the documentation and in Stackless/demo/tracing.py.

The implementation also changes slp_schedule_task_prepared / slp_restore_tracing to modify tracing related members of PyThreadState only with official API functions. This prevents a ref-counting problem and prevents incorrect values of the global flag _Py_TracingPossible in ceval.c

• Bitbucket: https://bitbucket.org/stackless-dev/stackless/issue/43

Originally reported by: Kristján Valur Jónsson (Bitbucket: krisvale, GitHub: kristjanvalur)

Writing a unittest for sending a sequence through a channel, I tried the obvious, but failed:

#!python
    def testSequence(self):
        def sender():
            self.c.send_sequence(xrange(10))
            self.c.close()
        data = []
        def receiver():
            for i in self.c:
                data.append(i)

        stackless.tasklet(sender)()
        stackless.tasklet(receiver)()
        stackless.run()
        self.assertEqual(data, range(10))
        self.assertTrue(self.c.closed)

This doesn't work, however. The receiver is stuck on a iter.next call at the end. This is because when the sender calls close, the channel balance is already -1. So it is set into 'closing' state, but that is it. In order to do this properly, the functions needed to be tweaked like this:

#!python
        def sender():
            self.c.send_sequence(xrange(10))
            self.c.close()
            # this needs to change, close does not wake up a receiver, we must pump it
            while self.c.closing and not self.c.closed:
                self.c.send(None)

        def receiver():
            for i in self.c:
                data.append(i)
            #remove the extra "pump" nones at the end....
            while data[-1] is None:
                data.pop(-1)
            data.append(10)

I wonder if this system is how we want to do it? Wouldn't it be nicer if close() were immediate? It would wake up any blocked tasklets right away? We would get rid of the 'closing' attribute, or make it synonomous with 'closed'.

As it is, I don't really se a use case for the sequence protocol if it needs to be shored up in boilerplate anyway.

Is there a use case for the current closing/closed state transfer?

• Bitbucket: https://bitbucket.org/stackless-dev/stackless/issue/53

Originally reported by: Anselm Kruis (Bitbucket: akruis, GitHub: akruis)

stackless.run() switches from the main tasklet to another tasklet T. If a schedule callback is invoked for this switch, the stackless.current is T. This is incorrect. Inspection of the code in slp_schedule_task_prepared() reveals, that Stackless calls the schedule callback before the switch.

I added a test case and I also have a preliminary fix.

• Bitbucket: https://bitbucket.org/stackless-dev/stackless/issue/42

Originally reported by: RMTEW FULL NAME (Bitbucket: rmtew, GitHub: rmtew)

(originally reported in Trac by @akruis on 2013-06-03 11:49:35)

Problem

If you run a tasklet only partially (the tasklet calls stackless.schedule_remove() ) and then pickle the tasklet for later execution, you do not want Python to kill the tasklet during PyThreadState_Clear.

The following test code demonstrates the problem:

import stackless
import threading

finally_run_count=0

def task():
    global finally_run_count
    try:
        stackless.schedule_remove(None)
    finally:
        finally_run_count += 1

def run():
    global tasklet
    t = stackless.tasklet(task)
    t()
    stackless.run()
    
    # if you comment the next line,
    # then t will be garbage collected 
    # at the end of this function. 
    tasklet = t

thread = threading.Thread(target=run)
thread.start()
thread.join()
tasklet = None

print "finally_run_count: %d" % (finally_run_count,)

This script emits "finally_run_count: 1". If you comment the line tasklet = t Python deletes the tasklet t during the execution of the thread. As a consequence the finally clause of function task won't be called and the script would print "finally_run_count: 0". Otherwise, the tasklet still exists when Python clears the thread state. In this case PyThreadState_Clear calls slp_kill_tasks_with_stacks which kills the tasklet.

Unfortunately it is not always easily possible to delete the tasklet in time. For instance an object that is part of a not yet collected reference cycle can reference the tasklet and delay its destruction. (This happened in our application flowGuide. Depending on the execution of the garbage collector a tasklet was killed or simply deleted.)

I propose to change the C-function slp_kill_tasks_with_stacks() to not kill a tasklet, if both conditions are met:

• The tasklet can be deleted (C-function tasklet_has_c_stack() returns 0.
• The tasklet is not scheduled (one of stackless.schedule_remove() or tasklet.remove() has been called).

Does this change make sense? Are there any pitfalls?

• Bitbucket: https://bitbucket.org/stackless-dev/stackless/issue/24

Originally reported by: Anselm Kruis (Bitbucket: akruis, GitHub: akruis)

It is a legal requirement to adhere to the PSF Trademark Usage Policy. Currently that is most certailly not the case.

I would like to update our Stackless specific documentation (rst-files, doc-strings and source code comments) to meet the conditions set by the PSF for usage of the word "python".

Plan:

1. Summarise the rules for using the word "python" in Stackless.

2. Update stackless specific *.rst files

3. Update stackless specific doc strings

4. Update other stackless specific documentation including sourcee code comments

PythonTrademarkUsage.odt.zip

• Bitbucket: https://bitbucket.org/stackless-dev/stackless/issue/34

The idea to share the stack switching code between different projects is not new, see issue #10. Last year Kristján Valur Jónsson made another attempt: pull request #230:

The Stackman project (http://github.com/kristjanvalur/stackman) aims to provide simple stack switching code for modern platforms. The aim is to have application-agnostic code that performs the basic functionality of saving machine registers and switching stack pointer, leaving the custom code of saving/restoring stack and choosing stack pointer to a user callback.

The resulting switching code is very succinct and easy to verify.

I open this issue to discuss those aspects, that are unrelated to a particular implementation.

What

Make it possible to use Stackman to implement stack-switching instead of the current Stackless code.

Why - Benefits and Drawbacks

argument \ stakeholder | Stackless maintainer | Python developer | end user ------------------------------ | ---------------------------- | ------------------------ | ----------- Stackman is simpler to maintain. | + | | Supporting a new platform / architecture / compiler does not require changes to Stackless, only to Stackman | | + | | Stackless own hard switching code works well, Stackman adds complexity to the overall project | - | | | Performance is better / worse | | | +/- |

Are there any arguments, why the performance of Stackman is better/worse than legacy Stackless stack switching?

When

Now. Stackless 3.7 is near its eol and Stackless 3.8 hasn't been released. Stackman seems to be sufficiently mature.

How

Pull request #230 proves that Stackman can be used, but there are many more ways to do it.

Options

1. Install Stackman as an external project. Add a make install option to Stackman. In Stackless add an option --with-stackman to configure.
2. Like option 1, but automatically detect Stackman and use an option to disable Stackman (--without-stackman).
3. Add Stackman as a git submodule to Stackless. That's what pull request #230 does.
4. Add a full copy of Stackman source (archive) to a sub-directory in the Stackless source code.
5. Add a partial copy of Stackman to the Stackless source code and automatically use it.

There might be more options.

Requirements

Some requirements. There might be more.

1. Using Stackman is a compile time option. It is still possible to use the legacy Stackless hard switching code.
2. No user visible changes in API, stability, usage, if Stackless uses Stackman.
3. Do not break a working build process, if building from a Stackless source archive. E.g. ./configure && make && make test && make install still works. Especially important for the Windows build process. As a consequence, Stackless should adhere to the Include-directory layout from C-Python.
4. If Stackman is not bundled with Stackless source (options 1 and 2), there should be a mechanism to detect, if the found/specified version of Stackman is compatible with Stackless.
5. No legal risk. Stackless currently is a leisure time project. (Currently I don't expect any problems here, it is just for completeness.)

Discussion

Just some preliminary thoughts

We can divide the possible options into two goups:

• option 1,2: Stackman is only available, if separately installed
• option 3...5: Stackman is always available

Currently Stackless is mostly used on Windows amd64 and Linux amd64 (32bit versions are fading away). On these platforms the existing hard switching code and building Stackless just works. Unless Stackman improves the performance, using Stackman or legacy Stackless stack-switching makes no difference. If there was a performance gain, we would prefer option 3...5.

Advantages of option 1 and 2:

• Implementation of option 1 or 2 is probably simpler than option 3 or 4. Adding a complete copy of the Stackman source causes its header files to live outside of ./Include. Therefore we would need to modify the installation process (make install and Windows Tools/msi) or the build process.
• Selecting the right library is a concern of the Stackman installation.
• Stackless source does not contain any precompiled binary.

Option 5 would be fairly similar to the legacy solution. Upgrading to a newer Stackman version become complicated.

Issue:

• pickle.load raises OverflowError if loaded content includes PyFrameObject, whose f_executing is set to SLP_FRAME_EXECUTING_INVALID, i.e. -1.

Solution:

• Use 'B' instead of 'b' to parse/build char value when frame object is pickled.

How to reproduce this issue:

• The following code causes OverflowError unexpectedly.

   import pickle
   import stackless
   import sys
  
  
   try:
       raise RuntimeError()
   except:
       traceback_object = sys.exc_info()[2]
   a = pickle.dumps(traceback_object)
   b = pickle.loads(a)
   c = pickle.dumps(b)
   d = pickle.loads(c)
   # => This raises "OverflowError: unsigned byte integer is less than minimum".
  

Details:

• f_executing in PyFrameObject is declared as char, which might be signed depending on implementation.
  Therefore, 'B' should be specified when Py_BuildValue and PyArg_ParseTuple is called.
• If 'b' is specified instead of 'B' and f_executing is set to SLP_FRAME_EXECUTING_INVALID, PyArg_ParseTuple in frame_setstate in Stackless\pickling\prickelpit.c raises OverflowError.
  This is raised in convertsimple in Python\getargs.c because SLP_FRAME_EXECUTING_INVALID is less than 0.

C-Python 3.8 adds various Audit Hooks, see. Various Stackless APIs could be used to bypaas those hooks. Therefore Stckless sould add appropriate calls of PySys_Audit() to relevant C-functions (mostly __reduce__() and __setstate__(), tasklet.frame, tasklet.cstate, cstack.\_\_str\_\_, cstack.startaddr, cframe.*, ).

C-Python Hooks to be considered

• code.__new__
• sys._current_frames
• sys._getframe
• sys.set_asyncgen_hooks_finalizer
• sys.set_asyncgen_hooks_firstiter

Stackless already creates the events sys.setprofile and sys.settrace

Commit 925dc7fb1d0 (branch main) / 8dbdf5f275 (branch 3.8) breaks test_pickle.TestAsyncGenAThrowPickling. The test fails with AssertionError: Tuples differ: ((ZeroDivisionError('bla bla'),), 2, True) != ((ZeroDivisionError('bla bla'),), 1, True). The difference is the direct effect of the upstream change.

The fix is trivial.

For now I can apply the fix to branch 3.8-slp. To apply it to branch main-slp, upstream commit 925dc7fb1d0 needs to be merged first.

This is the historical patch for issue #10 from Kristján Valur Jónsson.

I converted this patch from a backup of the original mercurial repository available from https://bitbucket-archive.softwareheritage.org/projects/kr/krisvale/stackless-tealet.html.