For a long time I have been requesting price and open interest data every single minute from the REST API. For open interest I have always been using https://fapi.binance.com/fapi/v1/openInterest?symbol=
I started by doing it for only a few markets, then maxing out to all spot and all USDM-futures markets (kline data) and openinterest for USDM-futures on top.
At some point I discovered that the process takes longer and longer (outside of normal spikes through users), eventually not finishing in under 1 minute. I reviewed the changes in the code, but nevertheless the delay remained. I changed my public IP, but no improvement.
As a last ressort I took my hardware and plugged it into a different ISP, again with different IP. Immediately the duration of the scraping went almost all the way back to “pre delay” times. No code changes, the very same scraping took place faster on the same hardware.
I solved this issue about a year ago by getting the kline data through websocket API instead of REST (although I was not in danger of hitting rate limits btw!!). After a few days my delay normalized again. I guessed I just got flagged by cloudflare or some binance internal api anti-spam (although again: I didn’t hit rate limits!). Through this measure I saved about 75% of get requests and was left with only open interest data on the REST endpoint.
Fast forward to now: Again the very same problem. Less than a day ago my open interest scraping went 1min+ to finish, without any changes in code. Since then I rebooted without success. I just tested again on different ISP and IP (on different hardware this time), and I conclude, that I am again somehow shadow banned.
This time I cannot put some of the scraping on to websocket.
I have searched for similar problems, without success. I find it hard to believe that nobody else had these issues before me.
EDIT: Technical details
The delay occurs in response time. Back in the day 98% of responses occured in <2s. During “artificial delay” the response times went 5s+ for each get request. Going more async didn’t solve the issue, it made it worse. Nowadays during normal period median request response time is still <2s. Right now it is 4s+ for me.
I timeout each request at 5s and request again instead (no timeout at all leaves me with about 0.x% of never getting a response). I think it is within margins to expect the api to respond in under 5s or else regard the request as failed and try again. During normal times response times are 99% <5s. During peaks it goes maybe to 90% <5s. I don’t think this is the issue, but I will test with 10s timeout just for the lulz.