Can I drop `first_update_id` and `last_update_id` while processing futures history data (BTCUSDT_T_DEPTH)?

Mithril · June 13, 2023, 2:38am

First, BTCUSDT_T_DEPTH_2020-07-01_depth_update.csv csv contain first_update_id and last_update_id , and last_update_id is used by manage local order book

The first processed event should have U <= lastUpdateId+1 AND u >= lastUpdateId+1.

So I wonder what shuold I do with first_update_id and last_update_id.

Second, what is the machenism behind first_update_id and last_update_id?
I find there are 1.12% rows which first_update_id != last_update_id, this may indicate some feature.

DATA:


	symbol	timestamp	first_update_id	last_update_id	side	update_type	price	qty	x
0	BTCUSDT	1593561600003	39406726156	39406726156	b	set	9133.00	2.088	1
2	BTCUSDT	1593561600014	39406726164	39406726164	a	set	9143.16	0.539	2
6	BTCUSDT	1593561600015	39406726168	39406726168	b	set	9130.90	6.000	3
7	BTCUSDT	1593561600018	39406726169	39406726169	a	set	9140.57	0.250	4
9	BTCUSDT	1593561600018	39406726171	39406726171	b	set	9135.90	0.807	5
...	...	...	...	...	...	...	...	...	...
35227106	BTCUSDT	1593647999914	39538108407	39538108407	a	set	9252.83	0.271	22255329
35227107	BTCUSDT	1593647999916	39538108413	39538108413	a	set	9252.83	0.542	22255330
35227108	BTCUSDT	1593647999919	39538108416	39538108416	b	set	9232.09	0.060	22255331
35227111	BTCUSDT	1593647999933	39538108444	39538108444	a	set	9236.04	0.852	22255332
35227113	BTCUSDT	1593647999983	39538108497	39538108497	a	set	9236.04	0.912	22255333

dino · June 14, 2023, 1:43am

where do you receive the csv file?

Mithril · June 20, 2023, 8:24am

Received from a friend, he let me to conduct research on this data.

ilammy · June 20, 2023, 9:33am

Your CSV data might be incomplete. Normally first & last update IDs in the depth streams are different.

For example, here are recent events on btcusdt@depth stream:

{"e":"depthUpdate","E":1687252901096,"s":"BTCUSDT","U":37433396486,"u":37433396619,"b":[["26766.88000000","12.52300000"],["26765.82000000","0.14943000"],["26765.56000000","0.00000000"],["26765.49000000","0.00000000"],["26764.33000000","0.02000000"],["26762.40000000","0.00000000"],["26762.35000000","0.20450000"],["26762.14000000","0.34073000"],["26761.34000000","0.00000000"],["26759.72000000","0.00000000"],["26759.47000000","0.34073000"],["26759.03000000","0.00000000"],["26758.24000000","0.00000000"],["26756.79000000","0.45430000"],["26751.30000000","0.00000000"],["26730.09000000","0.26674000"],["26712.39000000","0.01334000"],["26709.32000000","0.00000000"],["26708.78000000","0.15660000"],["26707.04000000","0.10000000"],["26702.65000000","0.01848000"],["26701.62000000","0.00000000"],["26699.44000000","0.00000000"],["26667.96000000","0.00000000"],["26629.04000000","0.00100000"],["26365.38000000","0.00378000"],["26151.00000000","15.37862000"],["25428.54000000","0.00000000"],["24087.39000000","0.00000000"],["16060.13000000","0.00000000"],["13383.44000000","0.00089000"],["13383.00000000","0.01911000"],["10000.00000000","177.09258800"]],"a":[["26766.89000000","4.80957000"],["26766.90000000","1.66985000"],["26766.91000000","0.32323000"],["26766.95000000","0.00000000"],["26766.97000000","1.77360000"],["26766.98000000","0.00000000"],["26767.05000000","0.00000000"],["26767.23000000","0.00000000"],["26767.36000000","0.00072000"],["26767.48000000","0.00000000"],["26768.20000000","0.37373000"],["26768.25000000","0.01294000"],["26768.71000000","0.00000000"],["26768.85000000","0.11278000"],["26769.79000000","0.20000000"],["26770.08000000","0.20000000"],["26770.52000000","0.22423000"],["26770.53000000","0.37370000"],["26770.91000000","0.00000000"],["26771.17000000","0.00000000"],["26772.04000000","0.09762000"],["26772.22000000","0.00000000"],["26772.60000000","0.34424000"],["26773.06000000","0.00000000"],["26774.00000000","0.00000000"],["26775.20000000","0.22424000"],["26775.56000000","0.06072000"],["26776.29000000","0.33240000"],["26778.44000000","0.20000000"],["26780.37000000","0.34073000"],["26780.62000000","0.00000000"],["26781.71000000","0.34073000"],["26781.96000000","0.00000000"],["26782.11000000","0.00000000"],["26782.85000000","2.20133000"],["26783.05000000","0.45430000"],["26783.30000000","0.00699000"],["26785.69000000","0.00373000"],["26785.88000000","0.01000000"],["26786.32000000","0.14000000"],["26789.13000000","0.20915000"],["26792.10000000","0.67261000"],["26792.63000000","0.00000000"],["26794.36000000","0.00000000"],["26794.56000000","0.00000000"],["26795.83000000","0.00000000"],["26800.00000000","6.57659000"],["26803.01000000","0.00000000"],["26806.10000000","0.01119000"],["26833.38000000","0.00100000"],["27053.60000000","0.38057000"],["27053.61000000","0.00000000"]]}
{"e":"depthUpdate","E":1687252902096,"s":"BTCUSDT","U":37433396620,"u":37433396674,"b":[["26766.88000000","12.51775000"],["26761.53000000","0.05686000"],["26759.94000000","0.09340000"],["26759.03000000","0.11558000"],["26754.17000000","0.01006000"],["26747.35000000","0.01862000"],["26744.23000000","0.00000000"],["26742.38000000","0.01798000"],["26739.76000000","0.00000000"],["26737.00000000","0.00000000"],["26735.52000000","0.01994000"],["26734.54000000","0.01831000"],["26733.60000000","0.01841000"],["26733.02000000","0.03585000"],["26732.89000000","0.00000000"],["26732.08000000","0.01841000"],["26731.91000000","0.00000000"],["26731.67000000","0.01836000"],["26730.97000000","0.00000000"],["26729.46000000","0.00000000"],["26729.34000000","0.00000000"],["26729.04000000","0.00000000"],["26485.14000000","0.56909000"],["16060.13000000","0.00000000"],["13383.44000000","0.00089000"]],"a":[["26766.89000000","4.66469000"],["26767.07000000","0.00000000"],["26767.08000000","0.00000000"],["26767.16000000","0.00000000"],["26769.09000000","0.03709000"],["26770.68000000","0.09340000"],["26771.17000000","0.00157000"],["26772.32000000","0.00652000"],["26775.36000000","0.06072000"],["26775.56000000","0.00000000"],["26778.44000000","0.10000000"],["26781.25000000","0.01500000"],["26790.53000000","0.00000000"],["27750.00000000","5.89146000"],["29443.57000000","0.00057000"]]}
{"e":"depthUpdate","E":1687252903096,"s":"BTCUSDT","U":37433396675,"u":37433396753,"b":[["26766.88000000","12.57082000"],["26765.55000000","0.01000000"],["26762.35000000","0.00000000"],["26762.14000000","0.00000000"],["26761.54000000","0.12006000"],["26760.99000000","0.00747000"],["26759.47000000","0.00000000"],["26758.26000000","0.00000000"],["26756.79000000","0.00000000"],["26756.18000000","0.01799000"],["26755.49000000","0.01785000"],["26755.17000000","0.01796000"],["26754.65000000","0.00000000"],["26754.01000000","0.00000000"],["26753.68000000","0.00000000"],["26751.28000000","0.00000000"],["26736.78000000","5.39787000"],["26735.96000000","0.00206000"],["26726.91000000","0.00000000"],["26707.73000000","0.03039000"],["26694.65000000","0.00257000"],["26560.63000000","0.00041000"],["26512.61000000","0.00042000"],["13383.44000000","0.00089000"],["13383.00000000","0.01911000"],["10000.00000000","177.09358800"],["9636.08000000","0.00125000"]],"a":[["26766.89000000","4.25978000"],["26766.90000000","1.02554000"],["26766.91000000","0.04109000"],["26767.12000000","0.00000000"],["26767.25000000","0.00000000"],["26767.52000000","0.00000000"],["26767.68000000","0.27146000"],["26767.81000000","0.03642000"],["26768.20000000","0.00000000"],["26768.22000000","0.01000000"],["26768.25000000","0.00000000"],["26768.40000000","0.37373000"],["26768.45000000","0.02074000"],["26768.85000000","0.00000000"],["26768.86000000","0.18782000"],["26769.09000000","0.00000000"],["26770.52000000","0.00000000"],["26770.53000000","0.00000000"],["26771.17000000","0.22426000"],["26772.32000000","0.00000000"],["26772.37000000","0.37367000"],["26773.45000000","0.03709000"],["26775.34000000","0.06072000"],["26780.37000000","0.00000000"],["26780.65000000","0.00000000"],["26781.71000000","0.00000000"],["26781.99000000","0.00000000"],["26783.05000000","0.00000000"],["26783.33000000","0.00000000"],["26833.43000000","0.50000000"],["26955.06000000","0.01486000"],["27167.89000000","0.00000000"],["27217.03000000","0.00081000"]]}

Observe how first update of each event (U) is last update of the previous event (u) + 1.

Every new event describes cumulative order book updates since the last event. If previous event had last update ID 100 and the next event corresponds to 5 updates, then the new event will have first update ID 101 and last update ID 105. Normally there is more than one order book update during 100 or 1000 ms interval, so first and last update IDs are usually different.

Update IDs should be used to synchronize <symbol>@depth WebSocket streams with the snapshots from /api/v3/depth REST endpoint. Additionally, if you want to avoid losing events, you should keep multiple connections open, in which case update IDs allow to synchronize and deduplicate the streams.