Magento 2 has been nothing but problems. With every update random stuff breaks and issues opened by people on project just get ignored or have no useful responses.

In my case the nginx proxy server was just dropping the app's response with the error:

upstream sent too big header while reading response header from upstream

Due to how a heroku/dokku app is deployed using a proxy it is difficult to debug errors between the two servers so I couldn't tell what header was actually causing the problem.

After messing with the fastcgi_buffer size in the magento apps config I discovered that the nginx proxy buffer must also be increased.

Luckily it is easy to set the proxy variables now without using a custom template:

dokku nginx:set app proxy-buffer-size 16k
dokku nginx:set app proxy-busy-buffer-size 32k
dokku nginx:set app proxy-buffers '8 32k'

Then rebuild the config with:

dokku proxy:build-config app

After updating the buffer sizes, it is clear that the content-security-policy-report-only header is way too large even though the CSP module is disabled.

So in summary, update both the apps fastcgi buffer and nginx's proxy buffer sizes.

I hope it helps someone.