Questions regarding building a new connector

Hi, I have some questions about building a new connector inside of the files:

1- What does this function “listen_for_order_book_diffs” do data_source file ?
2- What is the difference between “listen_for_order_book_diffs” and “listen_for_order_book_snapshots” ?
3- Do I need a web socket for the “listen_for_order_book_diffs” or it can be done through http requests?
4- When I was testing hummingbot on different connectors you guys do not list all market pairs of each connector does this mean my exchange pairs have to be listed by you inside of hummingbot? or is there way for me to list them or you just check whatever pairs I offer and then hummingbot uses them through the requests that happen when the user chooses my connector ?

Thanks in adavnce.

  1. It converts the unreliable order book diff data stream from the exchange API (web socket or not) to a unified message format (i.e. OrderBookMessage) and a reliable message queue. It’s there to reduce the amount of uncertainties that the order book tracker has to deal with. The order book tracker is the only responsible for processing those messages into accurate order books.

  2. In most exchanges, there are separate API endpoints (or websocket streams) for order book snapshots and diffs. Snapshots are usually a lot bigger than diffs because it contains the whole order book, while diffs only contains the differences between snapshots.

    The recommended way for market connectors is to use diffs for tracking order books primarily, but include snapshots once in a while (e.g. every 30 minutes) just in case the diff stream is not 100% reliable and we need to clean up any missed updates once in a while. Also, if the diff stream got disconnected (and thus we can expect we’ll be missing diff messages), a new snapshot should be pulled along with reconnecting to the diff stream. The order book data source is responsible for making that choice of how often and when to pull in a new snapshot. It should always output every messages from the diff stream.

  3. You don’t have to. But if the exchange’s API offers that option, it’s the best way to get a (semi-)reliable diff stream.

    On the other hand, there are some exchanges out there that don’t have a diff stream - every order book message they send out is a snapshot. In that case, you can just put all the messages to the snapshot stream and let listen_for_order_book_diffs() sit idle. Then you can write your order book tracker to just use the snapshot stream.

  4. When you configure hummingbot for trading, you’d usually specify some symbols for it to trade. If a set of symbols is specified, then the internal logic of hummingbot would configure the market connector to pay attention to only those symbols. By default, however, market connectors are able to track all trading pairs in an exchange. Say for example, if you look at the BinanceMarket connector - one of its constructor parameters is the symbols parameter. When you initialize a BinanceMarket with, say, ['ETHUSD'] - then the Binance connector will only track ETHUSD. But if you initialize it with an empty symbols set (which is the default), then it will track all the actively trading symbols in Binance.

1 Like

Great explanation,

Now I am building my new connector and I have created the data_source page which is similar to one of the already made connectors.

1- How can I test to see if it’s running properly?
2- How can I see my connector in the connector list when I am choosing one in the config?
3- In the last stop of the config you ask for rpc url, can I skip this part because I do this already in the auth file?
4- In my case because we have JS SDK implemented so I would make a docker container for it to run locally because we are non-custodial, is there a way for hummingbot to mention in its docs while using my connector after being approved of course to run docker container of my SDK at the same machine for it to work ?

I know those are some questions but I am sure a lot of developers would have the same questions and it would be good place for them to start, Thanks in advance :slight_smile:

Hey Mostafa,

  1. To test if the data_source is running correctly essentially means, whether you are able to build and maintain the order book. This means that the order book being maintained by your order_book_tracker is consistent with the live order book on the exchange.
    Now, there are several ways you can test the data_source and the order_book_tracker.
    First, there is using the Unit Tests we have in the test/integration folder, to be precise, you would be looking at the test_*_order_book_tracker.py. Although the examples we have are very preliminary, it is a good place to start especially if you are familiar with using a Debugger.
    Second option would be to use aioconsole. This method, although tedious, would be ideal when you want to debug/monitor the outputs of specific functions.
    Personally, I used a mixture of both when implementing the Bittrex connector. But if you’re accustomed to using a Debugger, I would suggest just using Unit Tests. Just to be thorough that my order book is consistent, I compared it with the order book on the exchange at certain time intervals.

  2. You would have to edit the following files client/hummingbot_application.py and client/settings.py. Refer to this PR for a quick overview of all the files that would require to be added/changed to implement an exchange connector. Note that Bittrex, is a Centralized Exchange.

  3. I don’t quite get your question here, but the RPC URL are mainly for Decentralized Exchanges(DEXes) and it should not be required to be in the auth class.

  4. In your integration PR, you could include additional information into a documentation page dedicated to your connector. That way users would be aware of the necessary steps for it to work. Refer to this folder for your reference.