Depth websockets keep closing (1006 but not clientside)

cnect · March 23, 2021, 4:34am

Using php + React/Ratchet to connect to the Depth websocket streams (any of them)
The same websocket code but for @trades work perfect, no disconnections at all.

When I subscribe to 340 symbols at 100ms (regardless if full updates, or level 5) the websocket never survives for more than a few minuntes.

I ran multiple Depth websockets independendly and they disconnect simultaneously (300ms difference) so it seems like something triggers that.

The error code is 1006 (abnormal disconnect), I tried to debug into the problem and all I could find was that the stream resource suddenly finishes.
I tried to minimize the code, so basically it’s an empty loop() that only responds lowlevel, same result.

I tried to subscribe to 1 symbol only, no disconnection happens.
When subscribing to the 340 symbols the disconnect happens from 150-180 seconds, when it happens it hits all Depth streams almost simultaneously.

I even tried to raw debug the incoming data, it just ends in the middle of a stream, no errors or anything.
(…crvusdt@depth@100ms",“data”:{“e”:“depthUpdate”,“E”:1616469614143,“s”:“CRVUSDT”,“U”:270409244,“u”:270409246,“b”:,“a”:[["2.38800)

I’m out of ideas by now.
I can ignore it and optimize my reconnect code but given that my trade stream with the same amount of symbols works flawless I have a hard time just giving up.

p-i · March 24, 2021, 3:49am

The only thing I can think of is a Buffering overrun - data is being buffered on Binance faster than it can be transmitted.

Look at the timestamps on packets being returned and see if you can spot an increasing gap between event time and receive time.

Also be sure to test your code without any “save to file” or “print to screen”, Just to make sure that it is not something on your system that is creating congestion on the Binance end.

Also, are you able to successfully work around the disconnects without losing packets?
What workaround(s) are you considering?

Forever grateful if you keep this thread updated with your progress (I also need to handle this task).

MJW · March 23, 2021, 5:01am

More details regarding how you subscribe from which URL

cnect · March 23, 2021, 3:05pm

I tried subscribing an empty stream live and I tried subscribing by URL, both with same results.
For example like that:
streams=btcusdt@depth@100ms/1inchbusd@depth@100ms/1inchusdt@depth@100ms/aavebusd@depth@100ms/aavedownusdt@depth@100ms/aaveupusdt@depth@100ms/aaveusdt@depth@100ms/acmbusd@depth@100ms/acmusdt@depth@100ms/adabusd@depth@100ms/adadownusdt@depth@100ms/adaupusdt@depth@100ms/adausdt@depth@100ms/aergobusd@depth@100ms/aionusdt@depth@100ms/akrousdt@depth@100ms/algobusd@depth@100ms/algousdt@depth@100ms/alphabusd@depth@100ms/alphausdt@depth@100ms/ankrusdt@depth@100ms/antbusd@depth@100ms/antusdt@depth@100ms/ardrusdt@depth@100ms/arpausdt@depth@100ms/asrusdt@depth@100ms/atmusdt@depth@100ms/atombusd@depth@100ms/atomusdt@depth@100ms/auctionbusd@depth@100ms/audbusd@depth@100ms/audiobusd@depth@100ms/audiousdt@depth@100ms/audusdt@depth@100ms/avabusd@depth@100ms/avausdt@depth@100ms/avaxbusd@depth@100ms/avaxusdt@depth@100ms/axsbusd@depth@100ms/axsusdt@depth@100ms/badgerbusd@depth@100ms/badgerusdt@depth@100ms/bakebusd@depth@100ms/balbusd@depth@100ms/balusdt@depth@100ms/bandbusd@depth@100ms/bandusdt@depth@100ms/batbusd@depth@100ms/batusdt@depth@100ms/bchabusd@depth@100ms/bchbusd@depth@100ms/bchusdt@depth@100ms/beamusdt@depth@100ms/belbusd@depth@100ms/belusdt@depth@100ms/blzusdt@depth@100ms/bnbbusd@depth@100ms/bnbdownusdt@depth@100ms/bnbupusdt@depth@100ms/bnbusdt@depth@100ms/bntbusd@depth@100ms/bntusdt@depth@100ms/btcbusd@depth@100ms/btcdownusdt@depth@100ms/btcstbusd@depth@100ms/btcstusdt@depth@100ms/btcupusdt@depth@100ms/btcusdt@depth@100ms/btsusdt@depth@100ms/bttbusd@depth@100ms/bttusdt@depth@100ms/busdusdt@depth@100ms/bzrxbusd@depth@100ms/bzrxusdt@depth@100ms/cakebusd@depth@100ms/cakeusdt@depth@100ms/celousdt@depth@100ms/celrusdt@depth@100ms/chrusdt@depth@100ms/chzusdt@depth@100ms/ckbbusd@depth@100ms/ckbusdt@depth@100ms/cocosusdt@depth@100ms/compbusd@depth@100ms/compusdt@depth@100ms/cosusdt@depth@100ms/cotiusdt@depth@100ms/coverbusd@depth@100ms/creambusd@depth@100ms/crvbusd@depth@100ms/crvusdt@depth@100ms/ctkbusd@depth@100ms/ctkusdt@depth@100ms/ctsibusd@depth@100ms/ctsiusdt@depth@100ms/ctxcusdt@depth@100ms/cvcusdt@depth@100ms/cvpbusd@depth@100ms/dashbusd@depth@100ms/dashusdt@depth@100ms/databusd@depth@100ms/datausdt@depth@100ms/dcrusdt@depth@100ms/dentusdt@depth@100ms/dexebusd@depth@100ms/dfbusd@depth@100ms/dgbbusd@depth@100ms/dgbusdt@depth@100ms/diabusd@depth@100ms/diausdt@depth@100ms/dntbusd@depth@100ms/dntusdt@depth@100ms/dockusdt@depth@100ms/dodobusd@depth@100ms/dodousdt@depth@100ms/dogebusd@depth@100ms/dogeusdt@depth@100ms/dotbusd@depth@100ms/dotdownusdt@depth@100ms/dotupusdt@depth@100ms/dotusdt@depth@100ms/drepusdt@depth@100ms/duskusdt@depth@100ms/egldbusd@depth@100ms/egldusdt@depth@100ms/enjbusd@depth@100ms/enjusdt@depth@100ms/eosbusd@depth@100ms/eosdownusdt@depth@100ms/eosupusdt@depth@100ms/eosusdt@depth@100ms/etcbusd@depth@100ms/etcusdt@depth@100ms/ethbusd@depth@100ms/ethdownusdt@depth@100ms/ethupusdt@depth@100ms/ethusdt@depth@100ms/eurbusd@depth@100ms/eurusdt@depth@100ms/fetusdt@depth@100ms/filbusd@depth@100ms/fildownusdt@depth@100ms/filupusdt@depth@100ms/filusdt@depth@100ms/fiobusd@depth@100ms/fiousdt@depth@100ms/firousdt@depth@100ms/flmbusd@depth@100ms/flmusdt@depth@100ms/forbusd@depth@100ms/frontbusd@depth@100ms/ftmusdt@depth@100ms/fttusdt@depth@100ms/funusdt@depth@100ms/fxsbusd@depth@100ms/gbpbusd@depth@100ms/gbpusdt@depth@100ms/ghstbusd@depth@100ms/grtbusd@depth@100ms/grtusdt@depth@100ms/gtousdt@depth@100ms/gxsusdt@depth@100ms/hardbusd@depth@100ms/hardusdt@depth@100ms/hbarbusd@depth@100ms/hbarusdt@depth@100ms/hegicbusd@depth@100ms/hiveusdt@depth@100ms/hntusdt@depth@100ms/hotusdt@depth@100ms/icxbusd@depth@100ms/icxusdt@depth@100ms/idexbusd@depth@100ms/injbusd@depth@100ms/injusdt@depth@100ms/iostbusd@depth@100ms/iostusdt@depth@100ms/iotabusd@depth@100ms/iotausdt@depth@100ms/iotxusdt@depth@100ms/iqbusd@depth@100ms/irisusdt@depth@100ms/jstbusd@depth@100ms/jstusdt@depth@100ms/juvbusd@depth@100ms/juvusdt@depth@100ms/kavausdt@depth@100ms/keyusdt@depth@100ms/kmdusdt@depth@100ms/kncbusd@depth@100ms/kncusdt@depth@100ms/kp3rbusd@depth@100ms/ksmbusd@depth@100ms/ksmusdt@depth@100ms/linkbusd@depth@100ms/linkdownusdt@depth@100ms/linkupusdt@depth@100ms/linkusdt@depth@100ms/litbusd@depth@100ms/litusdt@depth@100ms/lrcbusd@depth@100ms/lrcusdt@depth@100ms/lskusdt@depth@100ms/ltcbusd@depth@100ms/ltcdownusdt@depth@100ms/ltcupusdt@depth@100ms/ltcusdt@depth@100ms/ltousdt@depth@100ms/lunabusd@depth@100ms/lunausdt@depth@100ms/manabusd@depth@100ms/manausdt@depth@100ms/maticbusd@depth@100ms/maticusdt@depth@100ms/mblusdt@depth@100ms/mdtusdt@depth@100ms/mftusdt@depth@100ms/mithusdt@depth@100ms/mkrbusd@depth@100ms/mkrusdt@depth@100ms/mtlusdt@depth@100ms/nanobusd@depth@100ms/nanousdt@depth@100ms/nbsusdt@depth@100ms/nearbusd@depth@100ms/nearusdt@depth@100ms/neobusd@depth@100ms/neousdt@depth@100ms/nknusdt@depth@100ms/nmrbusd@depth@100ms/nmrusdt@depth@100ms/npxsusdt@depth@100ms/nulsusdt@depth@100ms/oceanbusd@depth@100ms/oceanusdt@depth@100ms/ognusdt@depth@100ms/ogusdt@depth@100ms/omgbusd@depth@100ms/omgusdt@depth@100ms/onebusd@depth@100ms/oneusdt@depth@100ms/ongusdt@depth@100ms/ontbusd@depth@100ms/ontusdt@depth@100ms/ornusdt@depth@100ms/oxtusdt@depth@100ms/paxbusd@depth@100ms/paxgusdt@depth@100ms/paxusdt@depth@100ms/perlusdt@depth@100ms/phabusd@depth@100ms/pntusdt@depth@100ms/prombusd@depth@100ms/psgbusd@depth@100ms/psgusdt@depth@100ms/qtumbusd@depth@100ms/qtumusdt@depth@100ms/reefbusd@depth@100ms/reefusdt@depth@100ms/renusdt@depth@100ms/repusdt@depth@100ms/rifusdt@depth@100ms/rlcusdt@depth@100ms/rosebusd@depth@100ms/roseusdt@depth@100ms/rsrbusd@depth@100ms/rsrusdt@depth@100ms/runebusd@depth@100ms/runeusdt@depth@100ms/rvnbusd@depth@100ms/rvnusdt@depth@100ms/sandbusd@depth@100ms/sandusdt@depth@100ms/scusdt@depth@100ms/sfpbusd@depth@100ms/sfpusdt@depth@100ms/sklbusd@depth@100ms/sklusdt@depth@100ms/snxbusd@depth@100ms/snxusdt@depth@100ms/solbusd@depth@100ms/solusdt@depth@100ms/srmbusd@depth@100ms/srmusdt@depth@100ms/stmxusdt@depth@100ms/storjusdt@depth@100ms/stptusdt@depth@100ms/straxbusd@depth@100ms/straxusdt@depth@100ms/stxusdt@depth@100ms/sunusdt@depth@100ms/susdusdt@depth@100ms/sushibusd@depth@100ms/sushidownusdt@depth@100ms/sushiupusdt@depth@100ms/sushiusdt@depth@100ms/swrvbusd@depth@100ms/sxpbusd@depth@100ms/sxpdownusdt@depth@100ms/sxpupusdt@depth@100ms/sxpusdt@depth@100ms/sysbusd@depth@100ms/tctusdt@depth@100ms/tfuelusdt@depth@100ms/thetausdt@depth@100ms/tomobusd@depth@100ms/tomousdt@depth@100ms/trbbusd@depth@100ms/trbusdt@depth@100ms/troyusdt@depth@100ms/trubusd@depth@100ms/truusdt@depth@100ms/trxbusd@depth@100ms/trxdownusdt@depth@100ms/trxupusdt@depth@100ms/trxusdt@depth@100ms/tusdbusd@depth@100ms/tusdusdt@depth@100ms/tvkbusd@depth@100ms/twtbusd@depth@100ms/twtusdt@depth@100ms/uftbusd@depth@100ms/umausdt@depth@100ms/unfibusd@depth@100ms/unfiusdt@depth@100ms/unibusd@depth@100ms/unidownusdt@depth@100ms/uniupusdt@depth@100ms/uniusdt@depth@100ms/usdcbusd@depth@100ms/usdcusdt@depth@100ms/utkusdt@depth@100ms/vetbusd@depth@100ms/vetusdt@depth@100ms/vidtbusd@depth@100ms/viteusdt@depth@100ms/vthousdt@depth@100ms/wanusdt@depth@100ms/wavesbusd@depth@100ms/wavesusdt@depth@100ms/wingbusd@depth@100ms/wingusdt@depth@100ms/winusdt@depth@100ms/wnxmusdt@depth@100ms/wrxbusd@depth@100ms/wrxusdt@depth@100ms/wtcusdt@depth@100ms/xemusdt@depth@100ms/xlmbusd@depth@100ms/xlmdownusdt@depth@100ms/xlmupusdt@depth@100ms/xlmusdt@depth@100ms/xmrbusd@depth@100ms/xmrusdt@depth@100ms/xrpbusd@depth@100ms/xrpdownusdt@depth@100ms/xrpupusdt@depth@100ms/xrpusdt@depth@100ms/xtzbusd@depth@100ms/xtzdownusdt@depth@100ms/xtzupusdt@depth@100ms/xtzusdt@depth@100ms/xvsbusd@depth@100ms/xvsusdt@depth@100ms/yfibusd@depth@100ms/yfidownusdt@depth@100ms/yfiibusd@depth@100ms/yfiiusdt@depth@100ms/yfiupusdt@depth@100ms/yfiusdt@depth@100ms/zecbusd@depth@100ms/zecusdt@depth@100ms/zenusdt@depth@100ms/zilbusd@depth@100ms/zilusdt@depth@100ms/zrxbusd@depth@100ms/zrxusdt@depth@100ms

MJW · March 23, 2021, 10:22pm

Is this for SPOT or Futures?
Subscribing so many streams from URL might hit some length limitation. Read this - https://binance-docs.github.io/apidocs/futures/en/#websocket-market-streams and try live subscription

cnect · March 24, 2021, 2:26am

It’s Spot!
It works fine to subscribe by URL and by live subscription. I tried both and I’ve had the exactly same results.
It works for 2-4 minutes and then it disconnects

MJW · March 24, 2021, 2:45am

Why do you have to subscribe so many streams with each stream pushing massive data? If your server cannot handle it, your session would be disconnected

cnect · April 4, 2021, 12:47am

MJW: It was indeed a flaw in the PHP React/Ratchet implementation (they don’t like threads), if the event loop can’t catch up it will just swallow events.

The reason why I subscribe for many is because I need the data.

Spectr_Lab · June 3, 2021, 9:17pm

RUS: Полностью разделяю проблемы CNECT. Моя проблема: Наблюдаю количество ответов с Binance в единицу времени 0,1 секунда, @depth5@100ms, больше чем запрошено пар. Превышение количества ответов может достигать более чем в 2 с лишним раза. Однако в режиме получения данных с интервалом 1 секунда, @depth5, такого не наблюдается ( или есть незначительное превышение). Вероятно это особенность формирования данных на стороне Binance и их буферизация перед выдачей потребителю, возможно интернет. Такое нестабильный трафик ответов может приводить к накоплению очереди в websockets(python) и нарушению ожидаемой или рассчитанной работы программы на pythone пользователя Binance. Желаю(предлагаю, требую) разработчикам API устранить это явление, т.к. в режиме получения данных с интервалом 1 секунда, @depth5, такого не наблюдается. Или дать свои комментарии и предложения по решению и обходу данной проблемы, в рамках применения websockets на python. Возможно именно такой неравномерный трафик и приводит к обрывам соединения, а также мешает нормальной работе моих скриптов.

Ниже приводится результаты подсчёта усредненного количество ответов за 0,5 сек по 72 парам. Хотелось бы иметь не более 72 ответов за отведенные 0,1 секунды. (<=72)

ENG: I fully agree with CNECT’s problems. My problem: I am observing the number of responses from Binance per unit of time 0.1 second, @ depth5 @ 100ms, more than the requested pairs. The excess of the number of responses can reach more than 2 times. However, in the mode of receiving data with an interval of 1 second, @ depth5, this is not observed (or there is a slight excess). This is probably a feature of the formation of data on the side of Binance and their buffering before issuing to the consumer, possibly the Internet. Such unstable response traffic can lead to a queue accumulation in websockets (python) and disruption of the expected or calculated program execution on the pythone of a Binance user. I wish (suggest, demand) API developers to eliminate this phenomenon, tk. in the mode of receiving data with an interval of 1 second, @ depth5, this is not observed. Or give your comments and suggestions on how to solve and work around this problem, within the framework of using websockets in python. Perhaps it is this uneven traffic that leads to connection breaks, and also interferes with the normal operation of my scripts.

Below are the results of calculating the average number of responses for 0.5 seconds for 72 pairs. I would like to have no more than 72 responses in the allotted 0.1 seconds. (<= 72)

USDT clear_count = 49.4 / num pair = 72
USDT clear_count = 103.8 / num pair = 72 ALARM!!!
USDT clear_count = 5.8 / num pair = 72
USDT clear_count = 51.8 / num pair = 72
USDT clear_count = 64.8 / num pair = 72
USDT clear_count = 33.6 / num pair = 72
USDT clear_count = 51.8 / num pair = 72
USDT clear_count = 34.6 / num pair = 72
USDT clear_count = 51.8 / num pair = 72
USDT clear_count = 92.4 / num pair = 72 ALARM!!!
USDT clear_count = 48.0 / num pair = 72
USDT clear_count = 34.4 / num pair = 72
USDT clear_count = 103.4 / num pair = 72 ALARM!!!
USDT clear_count = 75.4 / num pair = 72 ALARM!!!
USDT clear_count = 34.4 / num pair = 72
USDT clear_count = 68.8 / num pair = 72
USDT clear_count = 114.4 / num pair = 72 ALARM!!!
USDT clear_count = 31.8 / num pair = 72
USDT clear_count = 34.4 / num pair = 72
USDT clear_count = 47.2 / num pair = 72
USDT clear_count = 34.6 / num pair = 72
USDT clear_count = 34.6 / num pair = 72
USDT clear_count = 25.8 / num pair = 72
USDT clear_count = 25.6 / num pair = 72
USDT clear_count = 31.2 / num pair = 72
USDT clear_count = 171.6 / num pair = 72 ALARM!!!
USDT clear_count = 28.0 / num pair = 72
USDT clear_count = 35.8 / num pair = 72
USDT clear_count = 48.6 / num pair = 72
USDT clear_count = 25.4 / num pair = 72
USDT clear_count = 86.0 / num pair = 72 ALARM!!!
USDT clear_count = 52.2 / num pair = 72
USDT clear_count = 147.2 / num pair = 72 ALARM!!!
USDT clear_count = 25.6 / num pair = 72
USDT clear_count = 37.6 / num pair = 72
USDT clear_count = 220.2 / num pair = 72 ALARM!!! !!! <====== how and why? > 72!!!
USDT clear_count = 35.2 / num pair = 72
USDT clear_count = 35.8 / num pair = 72
USDT clear_count = 34.0 / num pair = 72

code = 1006 (connection closed abnormally [internal]), no reason
660 пар, 10 потоков, 120 пар в потоке к USDT, обрывы соединения именно по USDT и BUSD. ~5MBit/sec
660 pairs, 10 streams, 120 pairs per stream for USDT, connection drops for USDT and BUSD