Libtorrent v2.x memory limiting on linux

[mhertz]

No worries buddy, thanks for posting

[heydude]

Sorry to necro this, but I had looked in the libtorrent github issues and sure enough other people have hit this. Libtorrent does have a non-mmap posix compatibility mode, but the python bindings (at least what's available on Ubuntu 24.04 LTS) don't expose it.

I spent some time with Claude and came up with a small python C++ shim and patch to expose the setting. This completely resolved the memory issues for me, even with large torrents. There is probably a performance hit, but I haven't noticed anything significant.

Will just paste this here since they're small and future deluge users can find them easily.

lt_posix_disk_io.cpp

Code: Select all

/*
 * lt_posix_disk_io - Tiny Python extension to expose libtorrent's
 * posix_disk_io_constructor to Python.
 *
 * Build:
 *   sudo apt install libtorrent-rasterbar-dev libboost-python-dev
 *   ./build.sh
 *
 * Usage from Python:
 *   import libtorrent as lt
 *   from lt_posix_disk_io import set_posix_disk_io
 *   sp = lt.session_params(settings)
 *   set_posix_disk_io(sp)
 *   ses = lt.session(sp)
 */

#include <boost/python.hpp>
#include <libtorrent/session_params.hpp>
#include <libtorrent/disk_interface.hpp>
#include <libtorrent/posix_disk_io.hpp>

namespace lt = libtorrent;

void set_posix_disk_io(lt::session_params& params)
{
    params.disk_io_constructor = lt::posix_disk_io_constructor;
}

BOOST_PYTHON_MODULE(lt_posix_disk_io)
{
    boost::python::def("set_posix_disk_io", &set_posix_disk_io,
        "Set posix_disk_io_constructor on a session_params object.\n"
        "This uses read()/write() syscalls instead of mmap.");
}

build.sh

Code: Select all

#!/bin/bash
set -e

# Auto-detect Python version
PYVER=$(python3 -c "import sys; print(f'{sys.version_info.major}{sys.version_info.minor}')")
PYINCLUDE=$(python3 -c "import sysconfig; print(sysconfig.get_path('include'))")
PYLIBDIR=$(python3 -c "import sysconfig; print(sysconfig.get_config_var('LIBDIR'))")
EXT_SUFFIX=$(python3 -c "import sysconfig; print(sysconfig.get_config_var('EXT_SUFFIX'))")

echo "Python ${PYVER}: include=${PYINCLUDE}, ext=${EXT_SUFFIX}"

# Find the boost_python library name (e.g. boost_python312, boost_python3, boost_python)
BOOST_PYLIB=""
for candidate in "boost_python${PYVER}" "boost_python3" "boost_python"; do
    if ldconfig -p | grep -q "lib${candidate}\.so"; then
        BOOST_PYLIB="${candidate}"
        break
    fi
done

if [ -z "$BOOST_PYLIB" ]; then
    echo "ERROR: Could not find boost_python library. Install libboost-python-dev"
    exit 1
fi
echo "Using boost library: ${BOOST_PYLIB}"

# Build
g++ -shared -fPIC -O2 \
    -o "lt_posix_disk_io${EXT_SUFFIX}" \
    lt_posix_disk_io.cpp \
    -I"${PYINCLUDE}" \
    -l"${BOOST_PYLIB}" \
    -ltorrent-rasterbar

echo ""
echo "Built: lt_posix_disk_io${EXT_SUFFIX}"
echo ""
echo "Install with:"
echo "  sudo cp lt_posix_disk_io${EXT_SUFFIX} /usr/lib/python3/dist-packages/"

Patch, apply with cd /usr/lib/python3/dist-packages/ ; sudo patch -p1 < deluge_posix.patch

deluge_posix.patch

Code: Select all

--- a/deluge/core/core.py
+++ b/deluge/core/core.py
@@ -119,7 +119,17 @@
             'user_agent': user_agent,
             'ignore_resume_timestamps': True,
         }
-        self.session = lt.session(settings_pack, flags=0)
+        # Use posix_disk_io instead of mmap_disk_io to avoid excessive
+        # virtual memory usage and OOM kills under cgroup memory limits.
+        try:
+            from lt_posix_disk_io import set_posix_disk_io
+            session_params = lt.session_params(settings_pack)
+            set_posix_disk_io(session_params)
+            self.session = lt.session(session_params)
+            log.info('Session created with posix_disk_io')
+        except Exception as ex:
+            log.warning('Failed to set posix_disk_io, falling back to default: %s', ex)
+            self.session = lt.session(settings_pack, flags=0)
 
         # Load the settings, if available.
         self._load_session_state()