Attaching to a python app deployed using Dokku with gdb

August 07, 2020


Some notes on how to debug a python app deployed to a (development) server using dokku.


In this case we're using a conda buildpack which makes the process here much easier.


1. Enable ptrace calls by adding deploy docker-option with "--cap-add=SYS_PTRACE"


2. Restart the app


3. Start a shell sudo docker exec --privileged -it <container-name> bash


4. Install gdb with conda /path/to/miniconda/bin/conda install -c conda-forge gdb


5. Get the process id with ps -aux


6. Run gdb using the conda version of python /path/to/minconda/bin/gdb /path/to/minconda/bin/python <pid-to-attach-to>



Then debug away...


app# .heroku/miniconda/bin/gdb .heroku/miniconda/bin/python 9 
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from .heroku/miniconda/bin/python...
Attaching to program: /app/.heroku/miniconda/bin/python, process 9
[New LWP 204]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f6cc239cad3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84


If the issue is in a compiled extension just use bt

(gdb) bt
#0  0x00007f6cc239cad3 in epoll_wait ()
    at ../sysdeps/unix/syscall-template.S:84
#1  0x00007f6cc0a01175 in pyepoll_poll (self=0x7f6c99cf4cb0, 
    args=<optimized out>, kwds=<optimized out>)
    at /usr/local/src/conda/python-3.7.5/Modules/selectmodule.c:1562
#2  0x00005636b2a6ac34 in _PyMethodDef_RawFastCallKeywords (


otherwise use py-bt to get the python trace


(gdb) py-bt
Traceback (most recent call first):
  File "/app/.heroku/miniconda/lib/python3.7/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
  (frame information optimized out)
  (frame information optimized out)
  (frame information optimized out)
  File "/app/.heroku/miniconda/lib/python3.7/site-packages/tornado/ioloop.py", line 526, in run_sync
    self.start()
  (frame information optimized out)
  (frame information optimized out)
  File "main.py", line 19, in <module>
    main()
(gdb)


Happy debugging!