lag not closing down with aggregated trade stream

thomasd3 · August 30, 2021, 2:28pm

When subscribing to the aggregated trade stream (futures), I started to compare the event time to the time I receive the event.
I average this over a thousand trades and monitor all coins.

What I found is that if there is a network anomaly, the lag doesn’t close, it looks like you stay continuously behind once the anomaly has passed. Can anyone explain why the ones not received are not dropped and then we carry on with the fresh ones?

I wrote a tool to illustrate that (which also showed another problem with trade id sequences for which I opened another question), you can get the code at: https://gitlab.com/ThomasD3/binancemonitor

To test you can start it, pull the network cable and then put it back and you should see the lag being printed (it’s evaluated every 10s) and not closing.

tantialex · August 31, 2021, 6:42pm

Hi,

I tried replicating the issue you mentioned by using the code provided and below is my log.

It’s fair to say the result is working as expected and the highest amount of lag I experienced was roughly 2s.

dotnet run
subscribed to 132 instruments
31/08/2021 18:31:19 BTCUSDT    - ⚠️ missing trades, went from 1391092037 to 1391092039
31/08/2021 18:33:32 SOLUSDT    - ⚠️ average lag is 1.76s
31/08/2021 18:33:42 SOLUSDT    - ⚠️ average lag is 1.66s
31/08/2021 18:38:23 ⚠️ trades socket connection lost
31/08/2021 18:38:40 ⚠️ trades socket connection restored
31/08/2021 18:38:40 BTCUSDT    - ⚠️ missing aggregate trade id, went from  794316616 to  794318261
31/08/2021 18:38:40 MATICUSDT  - ⚠️ missing aggregate trade id, went from  137512353 to  137512748
31/08/2021 18:38:40 MTLUSDT    - ⚠️ missing aggregate trade id, went from   16526084 to   16528729
31/08/2021 18:38:40 ALGOUSDT   - ⚠️ missing aggregate trade id, went from   46188282 to   46188495
31/08/2021 18:38:40 SUSHIUSDT  - ⚠️ missing aggregate trade id, went from   94688269 to   94688478
31/08/2021 18:38:40 LITUSDT    - ⚠️ missing aggregate trade id, went from   16988606 to   16989825
31/08/2021 18:38:40 BATUSDT    - ⚠️ missing aggregate trade id, went from   31886077 to   31886237
31/08/2021 18:38:40 ANKRUSDT   - ⚠️ missing aggregate trade id, went from   31283517 to   31283617
31/08/2021 18:38:40 LINKUSDT   - ⚠️ missing aggregate trade id, went from  165910536 to  165911336
31/08/2021 18:38:40 KSMUSDT    - ⚠️ missing aggregate trade id, went from   53433547 to   53434654
31/08/2021 18:38:40 ALICEUSDT  - ⚠️ missing aggregate trade id, went from   50816009 to   50816531
31/08/2021 18:38:40 COMPUSDT   - ⚠️ missing aggregate trade id, went from   59457466 to   59457970
31/08/2021 18:38:40 ZILUSDT    - ⚠️ missing aggregate trade id, went from   42689491 to   42689558
31/08/2021 18:38:40 LINAUSDT   - ⚠️ missing aggregate trade id, went from   16920187 to   16920257
31/08/2021 18:38:40 EOSUSDT    - ⚠️ missing aggregate trade id, went from  107574802 to  107575311
31/08/2021 18:38:40 BNBUSDT    - ⚠️ missing aggregate trade id, went from  285365694 to  285366276
31/08/2021 18:38:40 FTMUSDT    - ⚠️ missing aggregate trade id, went from   69533506 to   69534877
31/08/2021 18:38:40 LRCUSDT    - ⚠️ missing aggregate trade id, went from   33156335 to   33156693
31/08/2021 18:38:40 ETHUSDT    - ⚠️ missing aggregate trade id, went from  531314347 to  531316169
31/08/2021 18:38:40 IOTXUSDT   - ⚠️ missing aggregate trade id, went from    3484320 to    3484384
31/08/2021 18:38:40 ADAUSDT    - ⚠️ missing aggregate trade id, went from  222129758 to  222130862
31/08/2021 18:38:40 DOGEUSDT   - ⚠️ missing aggregate trade id, went from  380558883 to  380559405
31/08/2021 18:38:40 BTTUSDT    - ⚠️ missing aggregate trade id, went from   20548281 to   20548387
31/08/2021 18:38:40 ETCUSDT    - ⚠️ missing aggregate trade id, went from  162352806 to  162353069
31/08/2021 18:38:40 AVAXUSDT   - ⚠️ missing aggregate trade id, went from   68000592 to   68002142
31/08/2021 18:38:40 TLMUSDT    - ⚠️ missing aggregate trade id, went from   10728111 to   10728366
31/08/2021 18:38:40 KEEPUSDT   - ⚠️ missing aggregate trade id, went from    3371639 to    3372239
31/08/2021 18:38:40 DOTUSDT    - ⚠️ missing aggregate trade id, went from  129103540 to  129105265
31/08/2021 18:38:40 NKNUSDT    - ⚠️ missing aggregate trade id, went from   18076258 to   18076362
31/08/2021 18:38:40 NEOUSDT    - ⚠️ missing aggregate trade id, went from   69985212 to   69985501
31/08/2021 18:38:40 FILUSDT    - ⚠️ missing aggregate trade id, went from  100633303 to  100633779
31/08/2021 18:38:40 ZENUSDT    - ⚠️ missing aggregate trade id, went from   36554981 to   36555205
31/08/2021 18:38:40 GRTUSDT    - ⚠️ missing aggregate trade id, went from   71202329 to   71202530
31/08/2021 18:38:40 AUDIOUSDT  - ⚠️ missing aggregate trade id, went from    1965050 to    1965276
31/08/2021 18:38:40 XMRUSDT    - ⚠️ missing aggregate trade id, went from   43727407 to   43727498
31/08/2021 18:38:40 BELUSDT    - ⚠️ missing aggregate trade id, went from   31737746 to   31737937
31/08/2021 18:38:40 ONEUSDT    - ⚠️ missing aggregate trade id, went from   26987237 to   26987714
31/08/2021 18:38:40 HBARUSDT   - ⚠️ missing aggregate trade id, went from   17076338 to   17076470
31/08/2021 18:38:40 C98USDT    - ⚠️ missing aggregate trade id, went from    9908504 to    9908759
31/08/2021 18:38:40 SOLUSDT    - ⚠️ missing aggregate trade id, went from   98746523 to   98750222
31/08/2021 18:38:40 BCHUSDT    - ⚠️ missing aggregate trade id, went from  144503832 to  144504085
31/08/2021 18:38:40 ATOMUSDT   - ⚠️ missing aggregate trade id, went from   62885608 to   62886324
31/08/2021 18:38:40 XLMUSDT    - ⚠️ missing aggregate trade id, went from  116728457 to  116728756
31/08/2021 18:38:40 WAVESUSDT  - ⚠️ missing aggregate trade id, went from   63226417 to   63226619
31/08/2021 18:38:40 QTUMUSDT   - ⚠️ missing aggregate trade id, went from   44696198 to   44696277
31/08/2021 18:38:40 SNXUSDT    - ⚠️ missing aggregate trade id, went from   43866418 to   43866562
31/08/2021 18:38:40 ALPHAUSDT  - ⚠️ missing aggregate trade id, went from   41737433 to   41738079
31/08/2021 18:38:40 RENUSDT    - ⚠️ missing aggregate trade id, went from   37006591 to   37006812
31/08/2021 18:38:40 CELRUSDT   - ⚠️ missing aggregate trade id, went from   10641931 to   10642079
31/08/2021 18:38:40 RAYUSDT    - ⚠️ missing aggregate trade id, went from    3441608 to    3442463
31/08/2021 18:38:40 IOSTUSDT   - ⚠️ missing aggregate trade id, went from   54469177 to   54469328
31/08/2021 18:38:40 OCEANUSDT  - ⚠️ missing aggregate trade id, went from   29924689 to   29924845
31/08/2021 18:38:40 RVNUSDT    - ⚠️ missing aggregate trade id, went from   19702391 to   19702460
31/08/2021 18:38:40 XTZUSDT    - ⚠️ missing aggregate trade id, went from   46222452 to   46222550
31/08/2021 18:38:40 SRMUSDT    - ⚠️ missing aggregate trade id, went from   44879710 to   44879994
31/08/2021 18:38:40 UNIUSDT    - ⚠️ missing aggregate trade id, went from  101397146 to  101397874
31/08/2021 18:38:40 BLZUSDT    - ⚠️ missing aggregate trade id, went from   29891612 to   29891711
31/08/2021 18:38:40 FLMUSDT    - ⚠️ missing aggregate trade id, went from   23928493 to   23928586
31/08/2021 18:38:40 REEFUSDT   - ⚠️ missing aggregate trade id, went from   31385305 to   31385392
31/08/2021 18:38:40 STORJUSDT  - ⚠️ missing aggregate trade id, went from   37557981 to   37558068
31/08/2021 18:38:40 XRPUSDT    - ⚠️ missing aggregate trade id, went from  203343792 to  203344867
31/08/2021 18:38:40 YFIIUSDT   - ⚠️ missing aggregate trade id, went from   46889879 to   46890164
31/08/2021 18:38:40 ZECUSDT    - ⚠️ missing aggregate trade id, went from   49443562 to   49443651
31/08/2021 18:38:40 NEARUSDT   - ⚠️ missing aggregate trade id, went from   37339877 to   37340059
31/08/2021 18:38:40 YFIUSDT    - ⚠️ missing aggregate trade id, went from   74824357 to   74824447
31/08/2021 18:38:40 CVCUSDT    - ⚠️ missing aggregate trade id, went from   26076282 to   26076416
31/08/2021 18:38:40 SCUSDT     - ⚠️ missing aggregate trade id, went from   14801671 to   14801770
31/08/2021 18:38:40 OGNUSDT    - ⚠️ missing aggregate trade id, went from   18982577 to   18982778
31/08/2021 18:38:40 TOMOUSDT   - ⚠️ missing aggregate trade id, went from   30235122 to   30235248
31/08/2021 18:38:40 SANDUSDT   - ⚠️ missing aggregate trade id, went from   56692523 to   56692788
31/08/2021 18:38:40 1INCHUSDT  - ⚠️ missing aggregate trade id, went from   66965599 to   66965877
31/08/2021 18:38:41 ICPUSDT    - ⚠️ missing aggregate trade id, went from   30364664 to   30364929
31/08/2021 18:38:41 BTCBUSD    - ⚠️ missing aggregate trade id, went from   20543717 to   20543984
31/08/2021 18:38:41 CHRUSDT    - ⚠️ missing aggregate trade id, went from   20672511 to   20672608
31/08/2021 18:38:41 IOTAUSDT   - ⚠️ missing aggregate trade id, went from   38129491 to   38129558
31/08/2021 18:38:41 BNBBUSD    - ⚠️ missing aggregate trade id, went from    2129616 to    2129680
31/08/2021 18:38:41 AAVEUSDT   - ⚠️ missing aggregate trade id, went from   76184512 to   76185045
31/08/2021 18:38:41 BTCUSDT_210924 - ⚠️ missing aggregate trade id, went from    2373907 to    2373945
31/08/2021 18:38:41 LTCUSDT    - ⚠️ missing aggregate trade id, went from  153380717 to  153381264
31/08/2021 18:38:41 DASHUSDT   - ⚠️ missing aggregate trade id, went from   49466032 to   49466271
31/08/2021 18:38:41 ONTUSDT    - ⚠️ missing aggregate trade id, went from   49720265 to   49720349
31/08/2021 18:38:41 BALUSDT    - ⚠️ missing aggregate trade id, went from   30757179 to   30757329
31/08/2021 18:38:41 RUNEUSDT   - ⚠️ missing aggregate trade id, went from   45111257 to   45111481
31/08/2021 18:38:41 AXSUSDT    - ⚠️ missing aggregate trade id, went from  106331281 to  106331385
31/08/2021 18:38:41 RLCUSDT    - ⚠️ missing aggregate trade id, went from   56504372 to   56504521
31/08/2021 18:38:41 RSRUSDT    - ⚠️ missing aggregate trade id, went from   47609108 to   47609250
31/08/2021 18:38:41 HOTUSDT    - ⚠️ missing aggregate trade id, went from   29296429 to   29296499
31/08/2021 18:38:41 CRVUSDT    - ⚠️ missing aggregate trade id, went from   34580659 to   34580877
31/08/2021 18:38:41 VETUSDT    - ⚠️ missing aggregate trade id, went from  101568536 to  101568776
31/08/2021 18:38:41 SXPUSDT    - ⚠️ missing aggregate trade id, went from  105794210 to  105794494
31/08/2021 18:38:41 BAKEUSDT   - ⚠️ missing aggregate trade id, went from   22389851 to   22390075
31/08/2021 18:38:41 ETHBUSD    - ⚠️ missing aggregate trade id, went from   14512956 to   14513218
31/08/2021 18:38:41 EGLDUSDT   - ⚠️ missing aggregate trade id, went from   39860456 to   39860692
31/08/2021 18:38:41 CHZUSDT    - ⚠️ missing aggregate trade id, went from   91850921 to   91851154
31/08/2021 18:38:41 UNFIUSDT   - ⚠️ missing aggregate trade id, went from   20880638 to   20880836
31/08/2021 18:38:41 DGBUSDT    - ⚠️ missing aggregate trade id, went from    6447389 to    6447488
31/08/2021 18:38:41 KAVAUSDT   - ⚠️ missing aggregate trade id, went from   57874247 to   57874597
31/08/2021 18:38:41 BZRXUSDT   - ⚠️ missing aggregate trade id, went from   26100937 to   26101007
31/08/2021 18:38:41 THETAUSDT  - ⚠️ missing aggregate trade id, went from   78432869 to   78432985
31/08/2021 18:38:41 SFPUSDT    - ⚠️ missing aggregate trade id, went from   20136639 to   20136743
31/08/2021 18:38:41 DODOUSDT   - ⚠️ missing aggregate trade id, went from   14341748 to   14341893
31/08/2021 18:38:41 AKROUSDT   - ⚠️ missing aggregate trade id, went from   17216544 to   17216588
31/08/2021 18:38:41 ADABUSD    - ⚠️ missing aggregate trade id, went from    2375236 to    2375319
31/08/2021 18:38:41 TRBUSDT    - ⚠️ missing aggregate trade id, went from   41391564 to   41391653
31/08/2021 18:38:41 MKRUSDT    - ⚠️ missing aggregate trade id, went from   43591398 to   43591472
31/08/2021 18:38:41 STMXUSDT   - ⚠️ missing aggregate trade id, went from   10449862 to   10449899
31/08/2021 18:38:41 SKLUSDT    - ⚠️ missing aggregate trade id, went from   34556465 to   34556610
31/08/2021 18:38:41 DEFIUSDT   - ⚠️ missing aggregate trade id, went from   10328240 to   10328275
31/08/2021 18:38:41 TRXUSDT    - ⚠️ missing aggregate trade id, went from   66829182 to   66829359
31/08/2021 18:38:41 ETHUSDT_210924 - ⚠️ missing aggregate trade id, went from    3575877 to    3575944
31/08/2021 18:38:41 ENJUSDT    - ⚠️ missing aggregate trade id, went from   62662818 to   62663025
31/08/2021 18:38:41 MANAUSDT   - ⚠️ missing aggregate trade id, went from   17928036 to   17928129
31/08/2021 18:38:41 CTKUSDT    - ⚠️ missing aggregate trade id, went from   19669900 to   19669993
31/08/2021 18:38:41 ICXUSDT    - ⚠️ missing aggregate trade id, went from   23735537 to   23735618
31/08/2021 18:38:41 OMGUSDT    - ⚠️ missing aggregate trade id, went from   67015378 to   67015877
31/08/2021 18:38:41 KNCUSDT    - ⚠️ missing aggregate trade id, went from   28688069 to   28688106
31/08/2021 18:38:41 ZRXUSDT    - ⚠️ missing aggregate trade id, went from   33921129 to   33921203
31/08/2021 18:38:41 1000SHIBUSDT - ⚠️ missing aggregate trade id, went from   44399795 to   44399892
31/08/2021 18:38:41 DOGEBUSD   - ⚠️ missing aggregate trade id, went from     529397 to     529474
31/08/2021 18:38:41 GTCUSDT    - ⚠️ missing aggregate trade id, went from    7350530 to    7350601
31/08/2021 18:38:42 LUNAUSDT   - ⚠️ missing aggregate trade id, went from   53129293 to   53129716
31/08/2021 18:38:42 COTIUSDT   - ⚠️ missing aggregate trade id, went from   21108180 to   21108296
31/08/2021 18:38:42 DENTUSDT   - ⚠️ missing aggregate trade id, went from   28663116 to   28663225
31/08/2021 18:38:43 XRPBUSD    - ⚠️ missing aggregate trade id, went from     837948 to     838058
31/08/2021 18:38:43 XEMUSDT    - ⚠️ missing aggregate trade id, went from   12543980 to   12544097
31/08/2021 18:38:43 MASKUSDT   - ⚠️ missing aggregate trade id, went from     926541 to     926659
31/08/2021 18:38:43 BANDUSDT   - ⚠️ missing aggregate trade id, went from   62772510 to   62772767
31/08/2021 18:38:44 ATAUSDT    - ⚠️ missing aggregate trade id, went from     108143 to     108269
31/08/2021 18:38:44 BTSUSDT    - ⚠️ missing aggregate trade id, went from   12781816 to   12781874
31/08/2021 18:38:45 HNTUSDT    - ⚠️ missing aggregate trade id, went from   33917615 to   33917736
31/08/2021 18:38:56 BTCDOMUSDT - ⚠️ missing aggregate trade id, went from    1120620 to    1120628

thomasd3 · September 1, 2021, 9:56am

There are a couple things to look at here:

First the structure of the data is a stream of aggregated trades. Each of this block has a sequential id. Inside each block is the first and last trade id it covers in the aggregation.
So, for example, an expected structure would be:

Agg trade id = 5; trade from 100 to 110
Agg trade id = 6; trade from 111 to 125
Agg trade id = 7; trade from 126 to 130
the first message “missing trades…” means that for two consecutive aggregated blocks (5, 6 and 7 in the example abve), the first trade of a block doesn’t match the last trade of the previous one +1, so the blocks were all received in order but some trades are not included in them. I opened another question to find out why (missing trades in the futures trade stream
the other messages “missing aggregate trade id”, just means that some data in the stream hasn’t been received.
when it comes to the lag, the center question here, the issue is the following: let’s say that you get block 2, 3, 4 on time, one chunk every 100ms, but then #5 is delayed. Now you’re 1second behind but then Binance presumably buffers a bit of data for you and will keep sending 6, 7, 8… but it will send at the same pace, like 100ms, so you will still receive data at the normal speed but then you’ll be forever 1s behind because Binance will not send you data 6, 7, 8 at high speed so that you catch up. The lag will persist for the lifetime of the connection. So that the only solution I found is to start a new subscription and cancel the previous one, with all the complications of merging the two streams on the fly during the overlap to not miss any data.

tantialex · September 1, 2021, 1:06pm

In regards to the lag; the server is not returning old-data, but rather the calculation of your average is incorrect, let me expand:

For starters, I reproduced the issue over two test cases, an exact replica of your code and a 3-symbol test. Logs for the small test case can be found at the bottom of this comment, however I did not include the exact replica due to comment character limits.

As you mentioned previously, your data structure is a KVP encapsulating Concurrent Queues in reference to a symbol. Each queue is set to be of size 1000.

Every-time a message is received from the server, you are pushing it to the queue, and dequeuing the earliest message to retain a queue of length 1000.

However, when you are calculating the average, at an interval for 10 seconds, you are considering all values within the queue, instead of the values obtained within that 10 second window. While both are considered acceptable approaches to calculating an average, the anomalies encountered during reconnects (receiving a message which was sent previous to disconnect, therefore having an lag of reconnectTime - disconnectTime) will skew the data until it is dequeued.

That being said, I made a few alterations in your code to drop the values after the average is calculated (every 10s) and ran the program again. This time I could see that the “lag spike” still occurred on reconnect, which is expected, however future averages all remained stable. Log for this test can be found at the bottom of this comment.

I hope this covers your question regarding the lag, let me know if you need further explanation.

Sampled Test Log

01/09/2021 11:44:00 subscribed to 3 instruments
01/09/2021 11:54:16 BTCUSDT    - ⚠️ average lag is 296.89s
01/09/2021 11:54:16 BNBUSDT    - ⚠️ average lag is 415.18s
01/09/2021 11:54:16 ⚠️ trades socket connection lost
01/09/2021 11:54:23 ⚠️ trades socket connection restored
01/09/2021 11:54:23 BNBUSDT    - ⚠️ missing aggregate trade id, went from  285743566 to  285749105
01/09/2021 11:54:23 BTCUSDT    - ⚠️ missing aggregate trade id, went from  795221641 to  795225564
01/09/2021 11:54:26 BTCUSDT    - ⚠️ average lag is 380.57s
01/09/2021 11:54:26 BNBUSDT    - ⚠️ average lag is 367.49s
01/09/2021 11:54:36 BTCUSDT    - ⚠️ average lag is 345.39s
01/09/2021 11:54:36 BNBUSDT    - ⚠️ average lag is 344.38s
01/09/2021 11:54:46 BTCUSDT    - ⚠️ average lag is 314.43s
01/09/2021 11:54:46 BNBUSDT    - ⚠️ average lag is 294.31s
01/09/2021 11:54:56 BTCUSDT    - ⚠️ average lag is 246.71s
01/09/2021 11:54:56 BNBUSDT    - ⚠️ average lag is 263.98s
01/09/2021 11:55:06 BTCUSDT    - ⚠️ average lag is 209.48s
01/09/2021 11:55:06 BNBUSDT    - ⚠️ average lag is 226.05s
01/09/2021 11:55:16 BTCUSDT    - ⚠️ average lag is 182.67s
01/09/2021 11:55:16 BNBUSDT    - ⚠️ average lag is 193.42s
01/09/2021 11:55:26 BTCUSDT    - ⚠️ average lag is 161.80s
01/09/2021 11:55:26 BNBUSDT    - ⚠️ average lag is 169.59s
01/09/2021 11:55:36 BTCUSDT    - ⚠️ average lag is 120.30s
01/09/2021 11:55:36 BNBUSDT    - ⚠️ average lag is 147.93s
01/09/2021 11:55:46 BTCUSDT    - ⚠️ average lag is 84.33s
01/09/2021 11:55:46 BNBUSDT    - ⚠️ average lag is 127.55s
01/09/2021 11:55:56 BTCUSDT    - ⚠️ average lag is 47.13s
01/09/2021 11:55:56 BNBUSDT    - ⚠️ average lag is 87.18s
01/09/2021 11:56:06 BTCUSDT    - ⚠️ average lag is 11.73s
01/09/2021 11:56:06 BNBUSDT    - ⚠️ average lag is 61.11s
01/09/2021 11:56:16 BNBUSDT    - ⚠️ average lag is 36.05s
01/09/2021 11:56:26 BNBUSDT    - ⚠️ average lag is 14.49s

Solution Test Log

01/09/2021 12:27:55 subscribed to 3 instruments
01/09/2021 12:28:04 BNBUSDT    - ⚠️ average lag is 0.14s | message count  146
01/09/2021 12:28:14 BNBUSDT    - ⚠️ average lag is 0.14s | message count   83
01/09/2021 12:28:24 BNBUSDT    - ⚠️ average lag is 0.14s | message count  215
01/09/2021 12:28:34 BNBUSDT    - ⚠️ average lag is 0.14s | message count  156
01/09/2021 12:28:44 BNBUSDT    - ⚠️ average lag is 0.14s | message count  223
01/09/2021 12:28:54 BNBUSDT    - ⚠️ average lag is 0.17s | message count  255
01/09/2021 12:29:04 BNBUSDT    - ⚠️ average lag is 0.14s | message count  319
01/09/2021 12:29:14 BNBUSDT    - ⚠️ average lag is 0.15s | message count  218
01/09/2021 12:29:24 BNBUSDT    - ⚠️ average lag is 0.14s | message count  140
01/09/2021 12:29:34 BNBUSDT    - ⚠️ average lag is 0.14s | message count  118
01/09/2021 12:29:44 BNBUSDT    - ⚠️ average lag is 0.38s | message count  175
01/09/2021 12:29:54 BNBUSDT    - ⚠️ average lag is 0.14s | message count  191
01/09/2021 12:30:04 BNBUSDT    - ⚠️ average lag is 0.23s | message count  195
01/09/2021 12:30:14 BNBUSDT    - ⚠️ average lag is 0.15s | message count  152
01/09/2021 12:30:24 BNBUSDT    - ⚠️ average lag is 0.14s | message count  157
01/09/2021 12:35:06 BNBUSDT    - ⚠️ average lag is 0.14s | message count   11
01/09/2021 12:36:28 ⚠️ trades socket connection lost
01/09/2021 12:36:34 ⚠️ trades socket connection restored
01/09/2021 12:36:35 BTCUSDT    - ⚠️ missing aggregate trade id, went from  795269953 to  795274695
01/09/2021 12:36:35 BNBUSDT    - ⚠️ missing aggregate trade id, went from  285780721 to  285785715
01/09/2021 12:36:36 BNBUSDT    - ⚠️ average lag is 0.14s | message count    8
01/09/2021 12:36:46 BNBUSDT    - ⚠️ average lag is 0.87s | message count   71
01/09/2021 12:36:56 BNBUSDT    - ⚠️ average lag is 0.14s | message count  106
01/09/2021 12:37:06 BNBUSDT    - ⚠️ average lag is 0.17s | message count  131
01/09/2021 12:37:16 BNBUSDT    - ⚠️ average lag is 0.14s | message count   84
01/09/2021 12:37:26 BNBUSDT    - ⚠️ average lag is 0.14s | message count   57
01/09/2021 12:45:53 BNBUSDT    - ⚠️ average lag is 45.43s | message count   11
01/09/2021 12:46:11 ⚠️ trades socket connection lost
01/09/2021 12:46:17 ⚠️ trades socket connection restored
01/09/2021 12:46:17 BTCUSDT    - ⚠️ missing aggregate trade id, went from  795275250 to  795281840
01/09/2021 12:46:17 BNBUSDT    - ⚠️ missing aggregate trade id, went from  285786182 to  285790630
01/09/2021 12:46:23 BNBUSDT    - ⚠️ average lag is 0.14s | message count   29
01/09/2021 12:46:33 BNBUSDT    - ⚠️ average lag is 0.14s | message count   70
01/09/2021 12:46:43 BNBUSDT    - ⚠️ average lag is 0.14s | message count  144
01/09/2021 12:46:53 BNBUSDT    - ⚠️ average lag is 0.16s | message count  105
01/09/2021 12:47:03 BNBUSDT    - ⚠️ average lag is 0.14s | message count   83
01/09/2021 12:47:13 BNBUSDT    - ⚠️ average lag is 0.14s | message count  108
01/09/2021 12:47:23 BNBUSDT    - ⚠️ average lag is 0.14s | message count  109
01/09/2021 12:47:33 BNBUSDT    - ⚠️ average lag is 0.14s | message count  130
01/09/2021 12:47:43 BNBUSDT    - ⚠️ average lag is 0.14s | message count  117
01/09/2021 12:47:53 BNBUSDT    - ⚠️ average lag is 0.14s | message count   79
01/09/2021 12:48:03 BNBUSDT    - ⚠️ average lag is 0.14s | message count   88
01/09/2021 12:48:13 BNBUSDT    - ⚠️ average lag is 0.14s | message count   88
01/09/2021 12:48:23 BNBUSDT    - ⚠️ average lag is 0.14s | message count   92
01/09/2021 12:48:33 BNBUSDT    - ⚠️ average lag is 0.14s | message count   60
01/09/2021 12:48:43 BNBUSDT    - ⚠️ average lag is 0.15s | message count   83
01/09/2021 12:48:53 BNBUSDT    - ⚠️ average lag is 0.14s | message count   65
01/09/2021 12:49:03 BNBUSDT    - ⚠️ average lag is 0.14s | message count   77
01/09/2021 12:49:13 BNBUSDT    - ⚠️ average lag is 0.14s | message count   67
01/09/2021 12:49:23 BNBUSDT    - ⚠️ average lag is 0.14s | message count   60
01/09/2021 12:49:33 BNBUSDT    - ⚠️ average lag is 0.14s | message count   65
01/09/2021 12:49:43 BNBUSDT    - ⚠️ average lag is 0.14s | message count   76