if self.live_data is already true, why am I still in the history?

[xyshell]

Hi ccxtbt,

I found something I don't understand while playing with ccxtbt. I think if self.live_data is true I should be getting the current data instead of still being in the history. Am I understanding it incorrectly? Some logs and codes to illustrate:

If self.live_data is not designed to provide this, and I'd like to determine if I'm at current data or still in the history, what's the suggested way to do so?

......
2022-01-16 19:00:00 < 2022-01-17 06:30:43.861385
2022-01-16 20:00:00 < 2022-01-17 06:30:43.862211
2022-01-16 21:00:00 < 2022-01-17 06:30:43.862751
2022-01-16 22:00:00 < 2022-01-17 06:30:43.863443
2022-01-16 23:00:00 < 2022-01-17 06:30:43.864055
2022-01-17 00:00:00 < 2022-01-17 06:30:43.864785
2022-01-17 00:30:44.886290 BTCUSDT Data Status: LIVE
2022-01-17 01:00:00 == 2022-01-17 06:30:44.886652   why still in the history when self.live_data is already true
2022-01-17 02:00:00 == 2022-01-17 06:30:45.833513   why still in the history when self.live_data is already true
2022-01-17 03:00:00 == 2022-01-17 06:30:46.081778   why still in the history when self.live_data is already true
2022-01-17 04:00:00 == 2022-01-17 06:30:47.166631   why still in the history when self.live_data is already true
2022-01-17 05:00:00 == 2022-01-17 06:30:47.680394   shouldn't self.live_data=true be at this line?

import datetime

import backtrader as bt
import ccxtbt


class TestStrategy(bt.Strategy):
    def next(self):
        if self.live_data:
            print(f"{self.data.datetime.datetime()} == {datetime.datetime.utcnow()}")
        else:
            print(f"{self.data.datetime.datetime()} < {datetime.datetime.utcnow()}")

    def notify_data(self, data, status, *args, **kwargs):
        dn = data._name
        dt = datetime.datetime.now()
        msg = 'Data Status: {}'.format(data._getstatusname(status))
        print(dt, dn, msg)
        if data._getstatusname(status) == 'LIVE':
            self.live_data = True
        else:
            self.live_data = False


cerebro = bt.Cerebro(quicknotify=True)
cerebro.addstrategy(TestStrategy)
store = ccxtbt.CCXTStore(exchange='binance', currency='USDT', retries=5, debug=False,
                         config={'enableRateLimit': True, 'options': {'defaultType': 'future'}})
cerebro.setbroker(store.getbroker())
cerebro.adddata(store.getdata(
    dataname=f'BTC/USDT', name=f'BTCUSDT', timeframe=bt.TimeFrame.Minutes,
    fromdate=datetime.datetime.utcnow()-datetime.timedelta(hours=1505),
    compression=60, ohlcv_limit=1505, drop_newest=True))
cerebro.run()

[munsunouk]

idk why you confusing between historical data and live data. because it is obvious that start from Live it is live and it is not historical data but live. when you see ccxt.feed code there is notification once
`

  def start(self, ):
          DataBase.start(self)
  
          if self.p.fromdate:
              self._state = self._ST_HISTORBACK
              self.put_notification(self.DELAYED)
              self._fetch_ohlcv(self.p.fromdate)
  
          else:
              self._state = self._ST_LIVE
              self.put_notification(self.LIVE)`

and load function has no notification so you can consider live as '=='

[kevenli]

I met the same situation in using "binancecoinm" exchange, and I found that the "ohlcv_limit" parameter whose default value is 20 can affect the LIVE notification.

To put is smaller, for example 5, after 4 bars recieved with mode DELAYED comes historical bars with mode LIVE.

Put it big enough can avoid meeting the historical data come early problem.

[ArmanBM]

same issue here using kucoin exchange !
at first, it prints out 'Live' signal but then the output shows historical data!!!
first line:

- 2022-04-22 12:05:42.718545 , Data Status: LIVE
others!: (look at the day)

`

• 2022-04-20 03:46:00
• 2022-04-20 03:47:00
• 2022-04-20 03:48:00
• 2022-04-20 03:49:00
• 2022-04-20 03:50:00

...
`

my code:

 class TestStrategy(bt.Strategy):
    def __init__(self):
        self.sma = bt.indicators.SMA(self.data, period=21)

    def next(self):
        if self.live_data:
            cash, value = self.broker.get_wallet_balance(Vls.currency1)
            print('cash, value',cash, value)
            if not self.position:
                print('lets buy somthing')
                self.order = self.buy()
                print('->',self.order)

        else:
            # Avoid checking the balance during a backfill. Otherwise, it will
            # Slow things down.
            cash = 'NA'

        for data in self.datas:
            print('{} - {} | Cash {} | O: {} H: {} L: {} C: {} V:{} SMA:{}'.format(data.datetime.datetime(),
                                                                                   data._name, cash, data.open[0],
                                                                                   data.high[0], data.low[0],
                                                                                   data.close[0], data.volume[0],
                                                                                   self.sma[0]))


def notify_data(self, data, status, *args, **kwargs):
        dn = data._name
        dt = datetime.now()
        msg = 'Data Status: {}'.format(data._getstatusname(status))
        print(dt, dn, msg)
        if data._getstatusname(status) == 'LIVE':
            self.live_data = True
            print('data is live now!')
        else:
            self.live_data = False

    def notify_order(self, order):
        print('nfo')
        print(f'in notify order: {order}')

cerebro = bt.Cerebro(quicknotify=True)

# Add the strategy
cerebro.addstrategy(TestStrategy)

# Create our store
config = {'apiKey': apikey,
          'secret': secret,
          'password': password,
          'enableRateLimit': True,
          'urls': {
              'referral': 'https://www.kucoin.com/?rcode=rPESSKX',
              'api': {'public': 'https://openapi-sandbox.kucoin.com', 'private': 'https://openapi-sandbox.kucoin.com', 'futuresPrivate': 'https://api-sandbox-futures.kucoin.com', 'futuresPublic': 'https://api-sandbox-futures.kucoin.com'},
              'test': {'public': 'https://openapi-sandbox.kucoin.com', 'private': 'https://openapi-sandbox.kucoin.com', 'futuresPrivate': 'https://api-sandbox-futures.kucoin.com', 'futuresPublic': 'https://api-sandbox-futures.kucoin.com'},
              'www': 'https://www.kucoin.com',
              'doc': ['https://docs.kucoin.com']}
          }
store = CCXTStore(exchange='kucoin', currency=Vls.currency1, config=config, retries=5, debug=False)
broker = store.getbroker()
cerebro.setbroker(broker)
data = store.getdata(dataname=Vls.dataname, name=Vls.dataname[0:3],
                     timeframe=bt.TimeFrame.Minutes,
                     compression=1, ohlcv_limit=45000, drop_newest=True) 

# Add the feed
cerebro.adddata(data)

# Run the strategy
cerebro.run()

[happydasch]

the issue happens because loading historical data stopped after the first call. i added a pull request with a fix for this.