In the last chapter working-with-assets, we introduced manually adding assets to trade, along with scanning where we hinted an issue scanning large market lists. Take the Binance spot market with well over 1000 symbols. Keeping true to our pro plan with 2 calls per second, will give us a scan time of 1000 assets / 2 calls per second = 500 seconds, which equals 8 minutes and 20 seconds. This is getting into the “unacceptable scan time territory”.
Shortlists
Again sticking to our 1h scans for the RSI and MACD reversal signals, we might have to shorten this list of 1000+ symbols. Lets imagine that this strategy is only relevant if the daily close price just crossed up and above the 30 period Exponential Moving Average (EMA30).
In this case, we can create a shortlist, for which we will scan our hourly RSI + MACD setups.
For this we will create a new bot that will run in parallel to our Phobos bot. We’ll give this a more generic name, one that might be relevant to other bots as well. We’ll call this, the shortlist-daily-ema30-cross bot!
Lets first review the strategy configuration:
shortlist:
id: "daily-ema30-cross" # The ID of the shortlist
emaPeriod: 30 # Extra custom parameter for the shortlist scanner
bot:
id: "shortlist-daily-ema30-cross" # The ID of the bot, notice that it's the same name as the shortlist ID, but prefixed with "shortlist-"
fsm:
isEnabled: false # Disable the FSM, as this bot only scans the market to create the shortlist
scanner:
isEnabled: true
assetDelay: 500 # How big a delay between assets in milliseconds
tick:
schedule: "1 1 0 * * *" # Run the scanner every day at 00:01:01
trading:
liveEnabled: falseHere, we’re adding some new items to the configuration:
- shortlist
- id: This is the id for which all items in the shortlist will be saved under. This shortlist will live in its own collection in MongoDB, under the name: <any-collection-prefix>shortlist-<shortlist-id>
- emaPeriod: We can add any configuration we want to both the config.yml and this strategy.yml, and this will be available in the config objects
These new configuration settings aren’t actually used by the framework itself. We’re simply adding this information here, as it belongs in configuration files. And we’re simply just using these configuration parameters in our scanner.
Rate limits
We’re getting into a small pickle here regarding rate-limits. Because notice that this daily shortlist scanner will run for 8 minutes and 20 seconds at 1 second past the first minute after midnight (00:01:01), but so is the Phobos scanner. For this we have a couple of solutions:
- We can reschedule our
Phobosbot to only run every 15 minutes:1 */15 * * * *, giving us enough time to complete the shortlist scan in between. - Upgrade to the expert plan, giving us 5 calls per second, instead of 2.
- Finally, we can create some custom exception in Phobos, telling it to ignore the FSM and Scanner ticks for 10 minutes at every midnight, however, this is out of the scope of this guide.
Lets go with the first option, and change this shortlist scanner tick schedule to 5 minutes past midnight: 1 5 0 * * *, which is when Phobos should be all done. Now, obviously, this depends how many assets the Phobos scanner is picking up.
Scanner
Our scanner implementation will look as follows:
// Create a new Scanner class
class Scanner {
// Create a constructor that accepts the config, database, utilities, taapiClient, order, and notifications objects
constructor(config, database, utilities, taapiClient, order, notifications) {
this.config = config;
this.database = database;
this.utilities = utilities;
this.taapiClient = taapiClient;
this.order = order;
this.notifications = notifications;
}
// Mandatory scan() function
async scan() {
// Get all pairs from the exchange through TAAPI
this.taapiClient.getNpmClient().getExchangeSymbols("crypto", this.config.exchange.id).then( async symbols => {
// Loop through each symbol
for(let s in symbols) {
// Get the symbol
let symbol = symbols[s];
// Reset bulk queires
this.taapiClient.resetBulkConstructs();
// Get latest closed candle
this.taapiClient.addCalculation("candle", symbol, "1d", "candle_1d", { backtrack: 1 });
// Add EMA30 calculation
this.taapiClient.addCalculation("ema", symbol, "1d", "ema30_1d", { backtrack: 1, period: this.config.shortlist.emaPeriod });
// Fetch all calculations
await this.taapiClient.executeBulk().then( async ta => {
// If debug mode is enabled, log the scan item
if(this.config.server.debugMode) {
console.log(`Examining ${s}:${symbol}...`);
}
// If the open price is less than the EMA30 and the close
// price is greater than the EMA30, then we have a crossover
if(ta.candle_1d.open < ta.ema30_1d.value && ta.candle_1d.close > ta.ema30_1d.value) {
// Add asset to shortlist
this.database.setShortlistItem(this.config.shortlist.id, symbol);
} else {
// Remove asset from shortlist
this.database.deleteShortlistItem(this.config.shortlist.id, symbol);
}
});
// Wait for the configured delay between each asset (rate-limits by TAAPI)
await this.utilities.sleep(this.config.bot.scanner.assetDelay);
};
});
}
}
module.exports = Scanner;With this, we’ve created a new bot, with its own scanner, simply to create a shortlist. Lets see how a bot can use this shortlist, instead of scanning an entire market.
For this we’ll create yet another bot, and we’ll name this Deimos, the second moon of Mars. Lets have a look at the configuration first:
bot:
id: "deimos"
fsm:
isEnabled: true
tradeDelay: 500
tick:
schedule: "1 */15 * * * *"
scanner:
isEnabled: true
assetDelay: 500
shortlist:
id: "daily-ema30-cross"
dbName: "strategies_shortlist-daily-ema30-cross" # Note we need the full DB name including all prefixes, as the configuration for another bot might differ from this bot!
tick:
schedule: "1 5 * * * *"
trading:
liveEnabled: falseSo we’re simply defining some properties for Deimos, and telling to run the FSM every 15 minutes on the hour, and the scanner 5 minutes past every hour.
Implementation of the Deimos scanner:
// Create a new Scanner class
class Scanner {
// Create a constructor that accepts the config, database, utilities, taapiClient, order, and notifications objects
constructor(config, database, utilities, taapiClient, order, notifications) {
this.config = config;
this.database = database;
this.utilities = utilities;
this.taapiClient = taapiClient;
this.order = order;
this.notifications = notifications;
}
// Mandatory scan() function
async scan() {
/**
* Fetch the shortlist using the configured shortlist ID and database name.
*
* The shortlist ID is used to identify the shortlist in the database. The next
* parameter is the query, which is used to filter the results. The third parameter
* is the sort, which is used to sort the results. The final parameter is the database
* name, which tells us that this shortlist is stored under a different bot.
*/
let shortlist = await this.database.getShortlist(this.config.bot.scanner.shortlist.id, {}, {}, this.config.bot.scanner.shortlist.dbName);
// Loop through each shortlist item
for(let s in shortlist) {
// Get the symbol
let symbolInfo = shortlist[s];
// Reset bulk queires
this.taapiClient.resetBulkConstructs();
// Add RSI calculation
this.taapiClient.addCalculation("rsi", symbolInfo.symbol, "1h", "rsi_1h");
// Add MACD calculation
this.taapiClient.addCalculation("macd", symbolInfo.symbol, "1h", "macd_1h");
// Fetch all calculations
await this.taapiClient.executeBulk().then( async ta => {
// If debug mode is enabled, log the scan item
if(this.config.server.debugMode) {
console.log(`Examining ${s}:${symbolInfo.symbol}...`);
}
// If the RSI is less than 30 and the MACD is greater than the MACD Signal
// then create the trade
if(ta.rsi_1h.value < 30 && ta.macd_1h.valueMACD > ta.macd_1h.valueMACDSignal) {
await this.database.createTrade(this.config.exchange.id, symbolInfo.symbol, "1h");
// Post a message to Slack
this.notifications.postSlackMessage(`Found trade: ${symbolInfo.symbol} on ${this.config.exchange.id} on 1h.`);
}
});
// Wait for the configured delay between each asset (rate-limits by TAAPI)
await this.utilities.sleep(this.config.bot.scanner.assetDelay);
}
this.notifications.postSlackMessage(`Scanning complete.`);
}
}
module.exports = Scanner;Note
It’s important to note that all the ‘shortlist’ configurations, aren’t in any place hardcoded. It’s simply a suggestion to place the shortlist config in some place. In fact, you could even make several shortlists within the same bot if that has a use-case within the same tick timing. When fetching and storing shortlist items, a shortlist ID will have to be provided to the method database.setShortlistItem(this.config.shortlist.id, symbol); So you may place this config where ever you see fit, or even skip it all together.
Conclusion
And with that, we’ve created a shortlist to shorten a list of symbols in the thousands, down to way fewer symbols. This allows us to very quickly scan through more complex TA, which additionally might be more time sensitive to get into a position sooner than later when a new hourly candle forms.
Lets now get into some hosting, as we’d really like to have our bot(s) running in the cloud and not our local machine.