[Alerting] event log is not "disabled" when initialization fails

[pmuellr]

When the event log ES initialization fails, the event log itself should be in a "disabled" state where it won't write event documents to the index. But it appears we've only done some of the work on this.

The code below is where the documents are indexed:

kibana/x-pack/plugins/event_log/server/event_logger.ts

Lines 184 to 189 in 3602f0f

	async function indexLogEventDoc(esContext: EsContext, doc: unknown) {
	esContext.logger.debug(`writing to event log: ${JSON.stringify(doc)}`);
	await esContext.waitTillReady();
	await esContext.esAdapter.indexDocument(doc);
	esContext.logger.debug(`writing to event log complete`);
	}

The waitTillReady() call actually returns a boolean indicating whether initialized completed successfully, but we don't check it. We should check the result and not write when it returns false.

The value returned by waitTillReady() is set here:

kibana/x-pack/plugins/event_log/server/es/context.ts

Lines 58 to 84 in 3602f0f

	initialize() {
	// only run the initialization method once
	if (this.initialized) return;
	this.initialized = true;
	this.logger.debug('initializing EsContext');
	setImmediate(async () => {
	try {
	await this._initialize();
	this.logger.debug('readySignal.signal(true)');
	this.readySignal.signal(true);
	} catch (err) {
	this.logger.debug('readySignal.signal(false)');
	this.readySignal.signal(false);
	}
	});
	}
	async waitTillReady(): Promise<boolean> {
	return await this.readySignal.wait();
	}
	private async _initialize() {
	await initializeEs(this);
	}
	}

It's only set to false when _initialize() throws an error, but the initializeEs() call in _initialize() (scroll down to the bottom of this snippet) can also return false when an error is detected.

Not immediately clear if we should even bother with these booleans set on the inner calls, when we could just throw an error instead of returning the boolean. Probably worth looking at them a little closer to see if that would be a simplification.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

I think there's a separate question of whether the event log search should also be "disabled" somehow when the the es initialization fails.

I think the answer is no. Some init failures may end up leaving a working event log index available, and in those cases, it would be nice to have access to whatever is in that log - never know, it may yield a clue as to why the init failed (but seems unlikely).

[mikecote]

Maybe a UI indicator as well