I m not versed with Pyhton refcounting and memory management but I have experienced some problems with av.VideoFrame

In my application I save numpy arrays to a video file using pyav. For this I call:

frame = av.VideoFrame(input_frame.width,input_frame.height,'bgr24') to create a frame for each image.

Since this is done inside a fn that is called continuously, I expect frame to go out of scope and be garbage collected. On MacOS this is the case and I don't have any trouble.

However on Linux I quickly run out of memory (I can watch the python process growing in memory use until the system hangs.)

A simple workaround is creating a single instance of frame attaching it to self. and re-using it by updating the planes.

Any ideas why this happens when I make new frames?

Overview

av==9.1.1 wheel released a couple of hours ago seems to be not packaged properly for Windows and Python 3.8 and 3.9.

Expected behavior

Import without issues.

Actual behavior

Traceback (most recent call last):
  File "C:\Users\circleci\project\test\datasets_utils.py", line 90, in _import
    module = importlib.import_module(package)
  File "C:\Users\circleci\project\env\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\circleci\project\env\lib\site-packages\av\__init__.py", line 44, in <module>
    from av._core import time_base, library_versions
ImportError: DLL load failed while importing _core: The specified module could not be found.

Investigation

CI runs:

• :heavy_check_mark: Python 3.7
• :heavy_multiplication_x: Python 3.8
• :heavy_multiplication_x: Python 3.9
• :heavy_check_mark: Python 3.10

Reproduction

import av

Versions

• OS: Windows
• PyAV runtime: 9.1.1

I was doing some decoding benchmarks with PyAV, and had a question about decoding in multiple threads. I'm trying to decode several mp4 files, each one in it's own thread. However, total CPU usage never exceeds 100%, it's almost as if PyAV has some sort of internal lock that doesn't allow more than one thread to decode at a time.

Does anyone understand why decoding multiple files with PyAV doesn't scale with multiple threads? I've looked through the code, but am not seeing anything super obvious. This is with Python 3.6, CPython on MacOS. The decode loop I'm running in each thread is very simple, basically:

def decode(input_file):
    container = av.open(input_file)
    for _ in container.decode(video=0):
        pass

def main():
    for input_file in sys.argv[1:]:
        t = threading.Thread(target=decode, args=(input_file,))
        t.start()

PS: I haven't tried experimenting with PyPy yet, but I wouldn't think CPython's GIL is a problem, since almost all the heavy lifting is being done in ffmpeg libraries, not Python code. Correct me if I'm wrong on this though...

Oh, am I impressed with how easy PyAV makes using ffmpeg! Thanks.

I am decoding video from a custom stream, I am managing to decode data in RAM and get image frames out. But the decoder wants to buffer data before giving me the first frame out. I would like to get each frame out as promptly as possible, and as I feed more data in, get out new frames--again as promptly as possibly.

Googling around there seems to be an ffmpeg option to control this: codec_ctx->delay (mentioned in http://ffmpeg.org/pipermail/libav-user/2014-December/007672.html).

Is that what I am looking for? Is there a way to set it via an av.open() option?

Thanks!

-kb

Hi,

We've been struggling (#445) with using libav to decode packets of compressed audio that come into our server via a dedicated websocket connection.

It appears that the only way to send data to libav is though some file name (which may actually be a URL of some sort). So our solution is currently to use a unix named pipe, av.open(pipe) a container, then read packets, frames, from the stream in the container. This worked OK-ish, until we hit #445.

## consume reads data from a named pipe, and returns the data through framesq.
## It should be called from a separate process. 
## Decompressed data is returned in a Queue framesq.
def consume(pipename, framesq, doneq, sr):
    try:
        container = av.open(pipename, "r", options={'max_delay': '10000'})
        audio = container.streams[0]

        resampler = av.AudioResampler(
            format = av.AudioFormat("s16p").packed,
            layout = int(1),
            rate = sr,
        )

        for packet in container.demux(audio):
            for frame in packet.decode():
                frame.pts = None
                for i, plane in enumerate(frame.planes):
                    frame = resampler.resample(frame)
                    bytes = frame.planes[0].to_bytes()
                    framesq.put(bytes)

    except Exception as e:
        logger.debug("Error in AV consumption: %r", e)

    doneq.put(True) ## signal end of data  

In the light of solution #325, I've been experimenting with UDP and unix domain sockets to set up the transport of data, to circumvent the problem that ffmpeg options don't seem to make much of a difference. However I can't get any of these to work (a boilerplate example would be appreciated:-).

But really, we would just want to send the actual compressed data (incl headers etc) to libav synchoneously, and receive the decompressed data out whenever libav has enough information to output another frame. But we can't find an entry to pyav / libav / ffmpeg do do such a thing, hence the workaround above.

Does anyone know if it would be possible to get a chunk-by-chunk implementation of the decoding?

This is a work in progress. The package builds and installs under py3, but it fails on import like so:

In [1]: import av
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-55f15964059f> in <module>()
----> 1 import av

/Users/danielallan/Documents/Repos/PyAV/av/__init__.py in <module>()
      8 
      9 # For convenience, IMPORT ALL OF THE THINGS (that are constructable by the user).
---> 10 from .audio.fifo import AudioFifo
     11 from .audio.format import AudioFormat
     12 from .audio.frame import AudioFrame

/Users/danielallan/container.pxd in init av.audio.fifo (/Users/danielallan/Documents/Repos/PyAV/src/av/audio/fifo.c:2743)()

/Users/danielallan/frame.pxd in init av.container (/Users/danielallan/Documents/Repos/PyAV/src/av/container.c:5922)()

/Users/danielallan/stream.pxd in init av.frame (/Users/danielallan/Documents/Repos/PyAV/src/av/frame.c:2108)()

/Users/danielallan/frame.pxd in init av.stream (/Users/danielallan/Documents/Repos/PyAV/src/av/stream.c:4773)()

/Users/danielallan/plane.pxd in init av.audio.frame (/Users/danielallan/Documents/Repos/PyAV/src/av/audio/frame.c:2564)()

ImportError: dlopen(/Users/danielallan/Documents/Repos/PyAV/av/plane.so, 2): Symbol not found: _PyString_FromStringAndSize
  Referenced from: /Users/danielallan/Documents/Repos/PyAV/av/plane.so
  Expected in: dynamic lookup

When using ffmpeg, we can use -hwaccel <hwtype> (for example with hwtype = cuvid) before -i to instruct ffmpeg to use the specified decoders, instead of the default one. In the case of cuvid, this would be h264_cuvid, hevc_cuvid, vp9_cuvid, etc.

This is really useful when we don't know in advance what codec a container uses, but we still want to use the hardware-accelerated decoders.

Is there a way to default to the hardware-accelerated decoders in PyAV?

Thanks!

• Add optional dictionary parameter to input open allowing the user to pass setting for hardware acceleration
• Add nvidia libraries to the ffmpeg build if available

Streams are not actually necessary for encoding/decoding, only muxing/demuxing. Refactor encoding/decoding onto a CodecContext, and then have a {Video,Audio,Subtitle}CodecContext which implements the specifics functions.

Do this for #154.

Overview

av/stream.c uses __pyx_v_c_stream->codec->codec_type, but it is deprecated ~4 years ago and will be removed in next ffmpeg version (libavformat>58).

In https://trac.ffmpeg.org/wiki/Bump59

... libavformat - AVFormatContext.open_cb replaced by io_open/io_close - AVStream.codec removed; stream information is now exported through AVStream.codecpar; for decoding/encoding the users should allocate a separate codec context - HTTP and RTSP protocol's 'user-agent' option renamed to 'user_agent' - ...

Expected behavior

pip install av --no-binary av works with the latest FFmpeg version

Actual behavior

Build failed due to the above compile error

Build report:

...
    src/av/stream.c: In function ‘__pyx_f_2av_6stream_wrap_stream’:
    src/av/stream.c:2257:27: error: ‘struct AVStream’ has no member named ‘codec’
     2257 |   switch (__pyx_v_c_stream->codec->codec_type) {
          |                           ^~
    src/av/stream.c: In function ‘__pyx_f_2av_6stream_6Stream__init’:
    src/av/stream.c:2875:29: error: ‘struct AVStream’ has no member named ‘codec’
     2875 |   __pyx_t_1 = __pyx_v_stream->codec;
          |                             ^~
    src/av/stream.c:2995:15: warning: assignment discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
     2995 |     __pyx_t_6 = __pyx_v_self->_codec_context->codec;
          |               ^
    src/av/stream.c: In function ‘__pyx_f_2av_6stream_6Stream__finalize_for_output’:
    src/av/stream.c:3688:130: error: ‘struct AVStream’ has no member named ‘codec’
     3688 |   __pyx_t_7 = __pyx_f_2av_5error_err_check(avcodec_parameters_from_context(__pyx_v_self->_stream->codecpar, __pyx_v_self->_stream->codec), 0, NULL); if (unlikely(__pyx_t_7 == ((int)-1))) __PYX_ERR(0, 144, __pyx_L1_error)
          |                                                                                                                                  ^~
    error: command '<here is my_gcc_path>' failed with exit code 1
 ...

Reproduction

build pyAV with latest ffmpeg library

Versions

• OS: Linux

• FFmpeg:

ffmpeg version N-102619-gebedd26eef Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 10.2.0 (GCC)
configuration: --prefix=<my path> --extra-cflags='-fPIC -m64' --enable-shared --disable-static --enable-gpl --enable-nonfree --enable-libvpx --enable-libmp3lame --enable-libopus --enable-libwebp --enable-libass --enable-cuda-nvcc --enable-libnpp --enable-cuvid --enable-nvenc --nvccflags='-gencode arch=compute_60,code=sm_60 -O2' --enable-libx264 --enable-libx265 --enable-libfdk-aac --enable-libwebp --enable-openssl --enable-libsvtav1 --enable-libvorbis --enable-fontconfig --enable-pic
libavutil      57.  0.100 / 57.  0.100
libavcodec     59.  1.100 / 59.  1.100
libavformat    59.  2.101 / 59.  2.101
libavdevice    59.  0.100 / 59.  0.100
libavfilter     8.  0.101 /  8.  0.101
libswscale      6.  0.100 /  6.  0.100
libswresample   4.  0.100 /  4.  0.100
libpostproc    56.  0.100 / 56.  0.100

Research

I have done the following:

• [x] Checked the PyAV documentation
• [x] Searched on Google
• [x] Searched on Stack Overflow
• [x] Looked through old GitHub issues
• [ ] Asked on PyAV Gitter
• [ ] ... and waited 72 hours for a response.

   We didn't find `avformat_open_input` in the libraries.
    We look for it only as a sanity check to make sure the build
    process is working as expected. It is not, so we must abort.

    Please open a ticket at https://github.com/mikeboers/PyAV/issues
    with the folowing information:

    PyAV: 0.3.1
    Python: 2.7.6 (default, Jun 22 2015, 17:58:13) \n[GCC 4.8.2]
    platform: Linux-3.19.0-59-generic-x86_64-with-Ubuntu-14.04-trusty
    extension_extra:
        libraries: ['X11', 'Xext', 'Xv', 'avcodec', 'avdevice', 'avfilter', 'avformat', 'avutil', 'bz2', 'm', 'swresample', 'swscale', 'xcb', 'xcb-render', 'xcb-shape', 'xcb-xfixes', 'z']
        library_dirs: ['/usr/local/lib']
        include_dirs: ['/usr/local/include', 'include', '/usr/include/python2.7']
    config_macros:
        PYAV_VERSION=0.3.1
        PYAV_VERSION_STR="0.3.1"
        PYAV_COMMIT_STR="unknown-commit"
        PYAV_HAVE_LIBSWRESAMPLE=1
        PYAV_HAVE_AV_CALLOC=1
        PYAV_HAVE_AV_FRAME_GET_BEST_EFFORT_TIMESTAMP=1

    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-A8DzRj/av/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-txMrAL-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-A8DzRj/av/

EDIT by @mikeboers: down the page I wrote:

I'm pretty sure there there is some sort of compilation error that arises during linking to the static library vs the dynamic ones. The "fix", I believe, is to discuss this in the error message, and expose a flag so that you can try again and print out all of the compiler errors.

Overview

Hi all. I'm experiencing some weird behavior when closing an InputContainer. When debugging with debugpy, I'll get a segmentation fault if I call container.close() or let the object go out of scope with a context manager. I've encountered this issue on different machines, OSes, vms, and it only seems to happen while debugging and after accessing data from a Stream object.

Expected behavior

InputContainer objects should be garbage collected with no segfault

Actual behavior

A SIGSEGV signal is emitted when an InputContainer is closed

Traceback:

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.2.0, pluggy-1.0.0
rootdir: /home/vm/Desktop/pyav-test
collected 1 item

test_av.py
Fatal Python error: Segmentation fault

Thread 0x00007ff86e5fb640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 324 in wait
  File "/usr/lib/python3.10/threading.py", line 607 in wait
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 261 in _on_run
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007ff86edfc640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 324 in wait
  File "/usr/lib/python3.10/threading.py", line 607 in wait
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 215 in _on_run
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007ff86f5fd640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 320 in wait
  File "/usr/lib/python3.10/threading.py", line 607 in wait
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_timeout.py", line 43 in _on_run
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap
Thread 0x00007ff86fdfe640 (most recent call first):
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 204 in _read_line
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 222 in _on_run
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007ff8705ff640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 324 in wait
  File "/usr/lib/python3.10/queue.py", line 180 in get
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 367 in _on_run
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Current thread 0x00007ff872668000 (most recent call first):
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_safe_repr.py", line 128 in _is_long_iter
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_safe_repr.py", line 112 in _repr
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_safe_repr.py", line 196 in _repr_iter
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_safe_repr.py", line 77 in __call__
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_xml.py", line 315 in get_variable_details
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_suspended_frames.py", line 51 in get_var_data
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 753 in internal_get_variable_json
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_constants.py", line 475 in new_func
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 527 in do_it
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 1772 in process_internal_commands
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2097 in _do_wait_suspend
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/pydevd.py", line 2062 in do_wait_suspend
  File "/home/vm/Desktop/pyav-test/test_av.py", line 11 in test_av
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/python.py", line 195 in pytest_pyfunc_call
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/python.py", line 1789 in runtest
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/runner.py", line 167 in pytest_runtest_call
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/runner.py", line 260 in <lambda>
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/runner.py", line 339 in from_call
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/runner.py", line 259 in call_runtest_hook
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/runner.py", line 220 in call_and_report
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/runner.py", line 131 in runtestprotocol
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/main.py", line 324 in _main
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/vm/Desktop/pyav-test/env/lib/python3.10/site-packages/_pytest/config/__init__.py", line 167 in main
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/testlauncher.py", line 36 in run
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/testlauncher.py", line 44 in <module>
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124 in _run_code
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135 in _run_module_code
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321 in run_path
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284 in run_file
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430 in main
  File "/home/vm/.vscode/extensions/ms-python.python-2022.20.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39 in <module>
  File "/usr/lib/python3.10/runpy.py", line 86 in _run_code
  File "/usr/lib/python3.10/runpy.py", line 196 in _run_module_as_main

Extension modules: _pydevd_bundle.pydevd_cython, _pydevd_frame_eval.pydevd_frame_evaluator, av._core, av.logging, av.bytesource, av.buffer, av.audio.format, av.enum, av.error, av.utils, av.option, av.descriptor, av.container.pyio, av.dictionary, av.format, av.stream, av.container.streams, av.sidedata.motionvectors, av.sidedata.sidedata, av.packet, av.container.input, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video.frame, av.video.stream, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.audio.fifo, av.filter.pad, av.filter.link, av.filter.context, av.filter.graph, av.filter.filter, av.audio.resampler, av.video.codeccontext (total: 44)

Investigation

I've been able to reproduce this behavior on the following platforms

• Windows 11 22H2
• Ubuntu 20.04 (WSL2)
• Ubuntu 22.04 (VM)
• MacOS Ventura 13.0.1

Each segfault happens after closing an InputContainer after accessing any of its Stream objects

Reproduction

The following code should reproduce the error

import av
import av.datasets

content = av.datasets.curated("pexels/time-lapse-video-of-night-sky-857195.mp4")
input_container = av.open(content)
video_stream = input_container.streams.get(video=0)
total_frames = video_stream.frames

input_container.close() # <--- segfault happens here

print(total_frames)

or

import av
import av.datasets

content = av.datasets.curated("pexels/time-lapse-video-of-night-sky-857195.mp4")
with av.open(content) as input_container:
    video_stream = input_container.streams.get(video=0)
    total_frames = video_stream.frames
    # segfault happens as this goes out of scope

print(total_frames)

Versions

• OS: Ubuntu 20.04 (WSL2)
• PyAV runtime:

PyAV v10.0.0
library configuration: --disable-static --enable-shared --libdir=/tmp/vendor/lib --prefix=/tmp/vendor --disable-alsa --disable-doc --disable-mediafoundation --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-lzma --enable-version3 --enable-zlib
library license: GPL version 3 or later
libavcodec     59. 37.100
libavdevice    59.  7.100
libavfilter     8. 44.100
libavformat    59. 27.100
libavutil      57. 28.100
libswresample   4.  7.100
libswscale      6.  7.100

• PyAV build:

pip installed

• FFmpeg:

ffmpeg version 4.2.7-0ubuntu0.1 Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
configuration: --prefix=/usr --extra-version=0ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil      56. 31.100 / 56. 31.100
libavcodec     58. 54.100 / 58. 54.100
libavformat    58. 29.100 / 58. 29.100
libavdevice    58.  8.100 / 58.  8.100
libavfilter     7. 57.100 /  7. 57.100
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  5.100 /  5.  5.100
libswresample   3.  5.100 /  3.  5.100
libpostproc    55.  5.100 / 55.  5.100

Research

I have done the following:

• [x] Checked the PyAV documentation
• [x] Searched on Google
• [x] Searched on Stack Overflow
• [x] Looked through old GitHub issues
• [ ] Asked on PyAV Gitter
• [x] ... and waited 72 hours for a response.

Additional context

Strangely enough, I've only encountered this issue while debugging. The code runs fine if no debugger is attached.

Overview

I want to have two functions:

• one to load a video array from an .mp4 file
• to store a video array to an .mp4 file

I tried implementing those as follows:

from ctypes import Union
from pathlib import Path
from typing import BinaryIO

import av
import numpy as np


def load(file_path, **kwargs) -> np.ndarray:
    frames = []
    with av.open(file_path, **kwargs) as container:
        for i, frame in enumerate(container.decode(video=0)):
            img = frame.to_image()
            frames.append(img)

    return np.moveaxis(np.stack(frames), -3, -2)


# following this approach: https://github.com/PyAV-Org/PyAV/issues/498
def save_to_file(
    np_tensor: np.ndarray,
    file_path: str,
    frame_rate: int = 30,
    codec: str = 'mpeg4',
) -> None:

    video_tensor = np.moveaxis(np.clip(np_tensor, 0, 255), -3, -2).astype('uint8')

    with av.open(file_path, mode='w') as container:
        if video_tensor.ndim == 3:
            video_tensor = np.expand_dims(video_tensor, axis=0)

        stream = container.add_stream(codec, rate=frame_rate)
        stream.width = video_tensor.shape[-2]
        stream.height = video_tensor.shape[-3]
        stream.pix_fmt = 'yuv420p'

        for vid in video_tensor:
            frame = av.VideoFrame.from_ndarray(vid, format="rgb24")
            frame.pts = None

            for packet in stream.encode(frame):
                container.mux(packet)

        for packet in stream.encode(None):
            container.mux(packet)


tmp_file = str(Path.cwd() / 'tmp.mp4')

video_tensor = np.array(
    [[
        [[1, 2, 3], [4, 5, 6]],
        [[7, 8, 9], [10, 11, 12]],
        [[13, 14, 15], [16, 17, 18]],
        [[19, 20, 21], [22, 23, 24]],
    ]]
)


save_to_file(np_tensor=video_tensor, file_path=tmp_file)

video_from_file = load(tmp_file)
assert video_tensor.shape == video_from_file.shape
print(f"tensor = {video_tensor}")
print(f"video_from_file = {video_from_file}")
assert np.allclose(video_tensor, video_from_file)

Expected behavior

I have an mp4 file, that I want to load to video_1. When storing the video_1 array to an mp4 file, and then load the video from that new file to a second array, video_2, I expect: np.allclose(video_1, video_2)

Actual behavior

I am getting an Assertion error:

tensor = [[[[ 1  2  3]
   [ 4  5  6]]

  [[ 7  8  9]
   [10 11 12]]

  [[13 14 15]
   [16 17 18]]

  [[19 20 21]
   [22 23 24]]]]
video_from_file = [[[[ 3  4  7]
   [ 3  4  7]]

  [[ 8  9 12]
   [ 8  9 12]]

  [[15 16 19]
   [15 16 19]]

  [[20 21 24]
   [20 21 24]]]]
Traceback (most recent call last):
  File "/Users/charlottegerhaher/Library/Application Support/JetBrains/PyCharm2022.2/scratches/scratch_26.py", line 68, in <module>
    assert np.allclose(video_tensor, video_from_file)
AssertionError

Traceback:

Traceback (most recent call last):
  File "/Users/charlottegerhaher/Library/Application Support/JetBrains/PyCharm2022.2/scratches/scratch_26.py", line 68, in <module>
    assert np.allclose(video_tensor, video_from_file)
AssertionError

Research

I have done the following:

• [x] Checked the PyAV documentation
• [x] Searched on Google
• [x] Searched on Stack Overflow
• [x] Looked through old GitHub issues
• [ ] Asked on PyAV Gitter
• [ ] ... and waited 72 hours for a response.

Overview

There's currently no .pyi files in the PyAV installation path, which brings much difficulty to the development, as the language server couldn't resolve any symbols and provides no completion or hints.

Desired Behavior

Add some basic stubs to PyAV, I'm not looking forward to a detailed type hint, just function signature and some documents, I'd heard that there's some tools like stubgen has the ability to finish the task automatically, but I'm not sure that it will work with Cython files and C module.

Additional context

I'll also try to write the stubs by myself, but I certainly need more help

Overview

• I want to traverse a video and after frame 10 - add 10 more frames made from current 10-nth frame onto which I draw an overlay and then resume and remux the rest of the frames.
• I don't know how to compute pts for new added frames and the rest

I am using the example provided by @jlaine at https://github.com/PyAV-Org/PyAV/issues/473 with great help to know how to compute pts.

I've came with this formula hardly digging

def frame_to_pts(frame, time_base, frame_rate, start_time):
    return int((frame + start_time * time_base * frame_rate) / (time_base * frame_rate))

then

import av
import cv2


# open input and find video stream
input_container = av.open('input.mp4', 'r')
input_stream = input_container.streams.get(video=0)[0]

# open output and add a video stream
output_container = av.open('output.mp4', 'w')
output_stream = output_container.add_stream('h264', rate=input_stream.rate)

frame_current = 0

for frame in input_container.decode(input_stream):
    # perform edge detection
    img = frame.to_ndarray(format='bgr24')
    img = cv2.cvtColor(cv2.Canny(img, 100, 200), cv2.COLOR_GRAY2BGR)

    if frame_current > 10 and frame_current <= 20:
        # add 10 more frames
        for i in range(10):
            # compute new pts
            pts_computed = frame_to_pts(frame_current, frame.time_base, input_stream.average_rate, input_stream.start_time)
            # rebuild a VideoFrame, preserving timing information
            new_frame = av.VideoFrame.from_ndarray(img, format='bgr24')
            new_frame.pts = pts_computed
            new_frame.time_base = frame.time_base
            # encode and mux
            for packet in output_stream.encode(new_frame):
                output_container.mux(packet)
            # advance one frame
            frame_current += 1
    else:
        # compute new pts
        pts_computed = frame_to_pts(frame_current, frame.time_base, input_stream.average_rate, input_stream.start_time)
        # rebuild a VideoFrame, preserving timing information
        old_frame = av.VideoFrame.from_ndarray(img, format='bgr24')
        old_frame.pts = pts_computed
        old_frame.time_base = frame.time_base
        # encode and mux
        for packet in output_stream.encode(old_frame):
            output_container.mux(packet)
            # advance one frame
        frame_current += 1


# flush and close output
for packet in output_stream.encode(None):
    output_container.mux(packet)
output_container.close()

Expected behavior

The formula above seems to work for h264 codec, but I have no confidence in it -

Actual behavior

Switching to h264_nvenc instead of h264 gets me pts < dts errors.

Investigation

Tried other codecs hevc_nvenc - seems to work

Research

I have done the following:

• [x] Checked the PyAV documentation
• [x] Searched on Google
• [x] Searched on Stack Overflow
• [x] Looked through old GitHub issues
• [x] Asked on PyAV Gitter
• [x] ... and waited 72 hours for a response.

fixes #1044

avformat_new_stream(AVFormatContext *s, const AVCodec *c) does not use their second parameter [1] So avcodec_parameters_from_context() should do the job [2]