Hey so I recently started a mini project and it’s been coming along slowly but surely. Anyway I have run into an issue I cannot figure out.
I am trying to keep a local record of the BTC-USDT orderbook. However after a little bit of time I notice that if the price goes down a little bit and then comes back up there are a couple ask prices that are not being removed. Essentially I have a couple ask prices that are lower than the highest or couple highest bid prices. This does not happen in the other direction IE, price goes up and comes back down; there are no bids higher than the best ask prices.
I also noticed that if the price goes down a bit, then up a bit (where there are a few asks that are lower than the best bid) then back down those prices that were lower than the best bid(s) do get removed but it’s a little too late. Again this doesn’t happen to the bid side (bid side always cleans up nicely) only the ask side. I also keep track of the UID and uID's to make sure I’m not missing any messages/updates.
Is this a known issue? I haven’t noticed it mentioned anywhere but should I just delete any asks that are lower than the best bid?
Just a little recap of what I’m doing (maybe I’m missing something? ) and as per the API documentation:
- I open a websocket to the diff. depth stream
- Buffer the incoming updates,
- Get an orderbook snapshot I get via REST
- First update applied has
U <= lastUpdateId +1 and u >= lastUpdateId +1
- As updates come in I make sure
U or first update ID of that aggregate stream/message is equal to u of the last message +1
- Apply these updates to my local orderbook (for both bids and asks) using the following rules (because the amount are absolute rather than relative):
- If the message price is in my orderbook and the amount is 0 I remove that price
- If the message price is in my orderbook but has an amount
!= 0 I update that price amount with the new price amount
- If the message price is NOT in my order book and has an amount
!=0 I add the price and amount to orderbook
- If the message price is NOT in my orderbook and the amount is
==0 I ignore it
Initially everything looks fine and works/updates fine for a little bit but again after about 30-40 seconds I notice that if the market price goes down a little and back up there are some asks that are not updated and the best bids are greater than the best asks which is not possible. I made sure all my messages are being applied and that the U ID’s of incoming messages are equal to u +1of the old messages so no messages are being lost.
The issues does not seem to happen in the other direction, if market price goes up and back down there are no bids that are higher than the best asks.
Also if the market price goes down, then up, then back down most of the time the asks that were not supposed to be there do get removed or replaced.
Is this a known issue or am I missing something? Should I just automatically remove any asks that are below the best bid after a bid passes that ask price?
Any help is greatly appreciated and I would like to thank you in advance.
Hello,
there’s a script about managing local orderbook that’s written python, can you maybe compare and check with https://github.com/binance-exchange/binance-toolbox-python/blob/master/spot/manage_local_order_book.py first, please?
Ok, so I just got a chance to look at this and my first thought is why did not find binance-connector or binance’s git hub page earlier, I spent over a month writing my own code to handle everything and now I feel so defeated as the code in your link is so short an elegant compared to mine, one of those big FML moments.
Next, I installed binance-connector set my API key as an environment variable (wasn’t sure what environment variable name to set my secret key to) but I cant connect I keep getting a WebSocket connection closed: connection was closed uncleanly (SSL error: certificate verify failed (in tls_process_server_certificate)), code: 1006, clean: False, reason: connection was closed uncleanly (SSL error: certificate verify failed (in tls_process_server_certificate)) error which I never came across before using my own code, however I don’t use an API key for authentication because the orderbook stream is publicly available without an API key. I couldn’t find a solution anywhere when I tried to search for it any idea why I may be getting this error? By the way I changed the endpoint to the production endpoint not the testnet.
Anyway the next question that came to mind, and intuition tells me there isn’t/shouldn’t be but I have to ask, is there a difference between the (production not testnet) publicly available API endpoints and a SPOT endpoint that requires an API key as far as orderbook goes? I am using the publicly available API endpoints and don’t provide an API key when retrieving data/subscribing to a websocket. Because orderbook or in fact every stream in the “websocket market streams”(https://binance-docs.github.io/apidocs/spot/en/#websocket-market-streams) is publicly available and not really a secret so no API key required an I’m confused as to why you need to provide one in the sample code, which them makes me think it’s a different endpoint/stream? Anyway if they retrieve their data from different places then it’s not a good comparison. to check if my code is working properly.
Hey! Thanks for the check and mentioning the API Key part in the script, it’s actually not necessary and the script has just been updated. Testing locally, it returns OK for me, can you retry again?
Hey so I really don’t know what I’m doing wrong I’ve spent literally all day yesterday and today trying to figure out what the issue is, went so far as to uninstall conda, delete any remaining files from the install including all my original environments, and any files left over, reinstall, recreate all my environments and I’m still having an issue. When I run my code I get no issues, it uses the standard requests library and the websockets-client library however when I try to run this I only get errors, the out put I get is
INFO:root:Connection with URL: wss://stream.binance.com:9443/ws INFO:root:Start to connect.... WARNING:root:WebSocket connection closed: connection was closed uncleanly (SSL error: certificate verify failed (in tls_process_server_certificate)), code: 1006, clean: False, reason: connection was closed uncleanly (SSL error: certificate verify failed (in tls_process_server_certificate)) ERROR:root:Lost connection to Server. Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly. ]. Retrying: 1 .... repeat the previosu message ...
I tried searching for a fix and if I get rid of the SSL error I just get a DNS error so no matter what I do I get nothing and I’m feeling pretty defeated as my code has no issues with SSL certification or DNS retrievals. Is there anything I need to setup prior to running this?
I have binance-connector library installed but what libraries does that use? Is there something I need to setup/change? It doesn’t look like your importing any other class files/modules besides the ones in the binance-connector library so I’m only using this code and not importing anything else, could that be a reason? I am really at a loss here so any help or sense of direction would be helpfull.
@LooseAllTheMonies Noticed you’ve opened another post in this forum. Let’s keep the conversation there if there’s further questions.
Hey I just wanted to let you know I got it working and I noticed because you are the creator/maintainer of the library that you would be interested in knowing that this script doesn’t exit cleanly on Windows because even if you throw a keyboard interrupt the websocket stays open and responds to server pings but doesn’t close entirely. This is due to, from what I understand, the exception being thrown during a task but not making it to the scope of the top level handler. I could be completely wrong about this but I ran into the same issue when doing a different project and the terminal just hangs with no way of exiting the script except for exiting the window, which is not ideal.
If I may suggest adding the following to take car of this :
if __name__ == '__main__':
try:
main()
except KeyboardInterrupt:
os._exit(0)
Hi @LooseAllTheMonies, I see you’ve also commented in the GitHub issue. Thanks for your contribution! I’ll pass the message around.