Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

App crashes with a TypeError when using the Redis instrumentation #2389

Closed
rowanmanning opened this issue Aug 19, 2024 · 6 comments · Fixed by #2397 or #2375 · May be fixed by #2420
Closed

App crashes with a TypeError when using the Redis instrumentation #2389

rowanmanning opened this issue Aug 19, 2024 · 6 comments · Fixed by #2397 or #2375 · May be fixed by #2420
Assignees
Labels
bug Something isn't working pkg:instrumentation-redis-4 priority:p1 Bugs which cause problems in end-user applications such as crashes, data inconsistencies

Comments

@rowanmanning
Copy link

What version of OpenTelemetry are you using?

  • @opentelemetry/sdk-node v0.52.1
  • @opentelemetry/auto-instrumentations-node v0.49.1

What version of Node are you using?

  • Node.js v20.16.0

What did you do?

@metart43 and I tried to use the redis package alongside Node.js auto-instrumentations. We're connecting to Redis with an empty string as a URL which the redis client accepts.

What did you expect to see?

We expected to see a log (Redis connected) and the app to continue to run.

What did you see instead?

The app crashes with a TypeError:

/redacted/node_modules/@opentelemetry/instrumentation-redis-4/build/src/utils.js:34
        diag.error('failed to sanitize redis connection url', err);
             ^

TypeError: Cannot read properties of undefined (reading 'error')
    at removeCredentialsFromDBConnectionStringAttribute (/redacted/node_modules/@opentelemetry/instrumentation-redis-4/build/src/utils.js:34:14)
    at getClientAttributes (/redacted/node_modules/@opentelemetry/instrumentation-redis-4/build/src/utils.js:11:65)
    at Commander.patchedConnect (/redacted/node_modules/@opentelemetry/instrumentation-redis-4/build/src/instrumentation.js:218:68)
    at Object.<anonymous> (/redacted/index.js:11:8)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Module._load (node:internal/modules/cjs/loader:1024:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:174:12)
    at node:internal/main/run_main_module:28:49

Additional context

@rowanmanning rowanmanning added the bug Something isn't working label Aug 19, 2024
@pichlermarc pichlermarc added priority:p1 Bugs which cause problems in end-user applications such as crashes, data inconsistencies pkg:instrumentation-redis-4 labels Aug 21, 2024
@trentm
Copy link
Contributor

trentm commented Aug 21, 2024

private _getPatchedClientConnect() {
const plugin = this;
return function connectWrapper(original: Function) {
return function patchedConnect(this: any): Promise<void> {
const options = this.options;
const attributes = getClientAttributes(this._diag, options);

That should be plugin._diag I think. this isn't the instrumentation class instance anymore here.
I'm not sure if the change to this._diag was a recent change?

@trentm
Copy link
Contributor

trentm commented Aug 22, 2024

@rowanmanning I see how the error can happen and I'll come up with a fix. However, to reproduce this, I'm really curious what your Redis client config (or some other usage I'm missing) resulted in hitting this code path. The crash is because the catch here is being called:

try {
const u = new URL(url);
u.searchParams.delete('user_pwd');
u.username = '';
u.password = '';
return u.href;
} catch (err) {
diag.error('failed to sanitize redis connection url', err);
}

What's in your Redis client options.url that is triggering this, I wonder.
Are you able to tweak /redacted/node_modules/@opentelemetry/instrumentation-redis-4/build/src/utils.js by adding something like:

console.log('the err:', err);

On line 32, just before the diag.error(...) call?

@metart43
Copy link

Sorry to jump-in here, @trentm.

I think in the example @rowanmanning provided the value of redisUrl is an empty string. I haven't run the GIST but it should help you to replicate the error. You'd need to initialize Redis client with redisUrl being an empty string.
I put the console.log inside the catch block - The error I've gotten back from the OTEL package

error from OTEL {
  err: TypeError [ERR_INVALID_URL]: Invalid URL
      at new NodeError (node:internal/errors:405:5)
      at new URL (node:internal/url:611:13)
      at removeCredentialsFromDBConnectionStringAttribute (/path/to/your/node_modules/@opentelemetry/instrumentation-redis-4/build/src/utils.js:27:19)
      at getClientAttributes (/path/to/your/node_modules/@opentelemetry/instrumentation-redis-4/build/src/utils.js:11:65)
      at Commander.patchedConnect (/path/to/your/node_modules/@opentelemetry/instrumentation-redis-4/build/src/instrumentation.js:218:68)
      at Object.<anonymous> (/path/to/your/project/server/lib/redis-client.js:40:8)
      at Module._compile (node:internal/modules/cjs/loader:1256:14)
      at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
      at Module.load (node:internal/modules/cjs/loader:1119:32)
      at Module._load (node:internal/modules/cjs/loader:960:12) {
    input: '',
    code: 'ERR_INVALID_URL'
  }
}

@rowanmanning
Copy link
Author

rowanmanning commented Aug 22, 2024

Hi @trentm, thanks for jumping on this 🙂

If I add in a log before the diag.error as you suggested I get the same as Artem:

TypeError: Invalid URL
    at new URL (node:internal/url:797:36)
    at removeCredentialsFromDBConnectionStringAttribute (/redacted/node_modules/@opentelemetry/instrumentation-redis-4/build/src/utils.js:27:19)
    at getClientAttributes (/redacted/node_modules/@opentelemetry/instrumentation-redis-4/build/src/utils.js:11:65)
    at Commander.patchedConnect (/redacted/node_modules/@opentelemetry/instrumentation-redis-4/build/src/instrumentation.js:218:68)
    at Object.<anonymous> (/redacted/index.js:11:8)
    at Module._compile (node:internal/modules/cjs/loader:1358:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1416:10)
    at Module.load (node:internal/modules/cjs/loader:1208:32)
    at Module._load (node:internal/modules/cjs/loader:1024:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:174:12)

The url value is an empty string and you can replicate this error by running:

node -e "new URL('')"

I'm really curious what your Redis client config (or some other usage I'm missing) resulted in hitting this code path...

We're adding OpenTelemetry to quite an old app which made some interesting decisions in the past that we can't really unpick easily. The URL is set like this in the real app:

const redisUrl = process.env.REDIS_TLS_URL || process.env.REDIS_URL || '';
const redisOptions = { /* blah blah */ };

const client = redis.createClient({ url: redisUrl, ...redisOptions });

Parts of the app can run without Redis, we seem to be relying on the existing behaviour of redis.createClient which doesn't throw if we pass an empty string as the URL. It does log some errors but the non redis-based routes still work fine.

@trentm
Copy link
Contributor

trentm commented Aug 22, 2024

I think in the example @rowanmanning provided the value of redisUrl is an empty string.

and

The url value is an empty string and you can replicate this error by running:

Thanks both! My bad for not reading the whole (excellent) bug report. I quickly jumped straight to the code when I saw the error stack. :)

repro, and why the OTel code has an error

I can repro now.

A thing that had been confusing me was how the redis package was okay with the options.url, but the OTel JS code wasn't, because they are both using new URL(options.url). The difference is that redis has this guard:

https://github.com/redis/node-redis/blob/2fc79bdfb375602e2aaba15962f57c88d8afe46b/packages/client/lib/client/index.ts#L242-L243

but the OTel code has this guard:

if (typeof url !== 'string') {
return;
}
try {
const u = new URL(url);

log spam

While the crash is (will be, once there is a release) now fixed. There is still the log spam.
Using this repro:

use-redis.js

const { diag, DiagConsoleLogger, DiagLogLevel } = require('@opentelemetry/api');
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.INFO);

const path = require('path');
const {NodeSDK} = require('@opentelemetry/sdk-node');
const {RedisInstrumentation} = require('./'); // This is `@opentelemetry/instrumentation-redis-4`, I was working in a git clone.
const sdk = new NodeSDK({
    serviceName: path.parse(process.argv[1]).name,
    instrumentations: [
        new RedisInstrumentation()
    ],
});
process.once('beforeExit', async () => {
  sdk.shutdown();
});
sdk.start();

const otel = require('@opentelemetry/api');
const redis = require('redis');

async function main() {
    const client = redis.createClient({
        url: '',
        socket: {
            port: '6379',
            host: process.env.REDIS_HOST,
        },
    });
    await client.connect();
    await client.set('foo', 'bar');
    const val = await client.get('bar');
    console.log('GET bar:', val);
    await client.quit();
}

const tracer = otel.trace.getTracer('test');
tracer.startActiveSpan('manual-parent-span', async (span) => {
    await main();
    span.end();
});
Log spammy output
% node use-redis.js
OTEL_TRACES_EXPORTER is empty. Using default otlp exporter.
@opentelemetry/instrumentation-redis-4 failed to sanitize redis connection url TypeError [ERR_INVALID_URL]: Invalid URL
    at new NodeError (node:internal/errors:405:5)
    at new URL (node:internal/url:676:13)
    at removeCredentialsFromDBConnectionStringAttribute (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/build/src/utils.js:27:19)
    at getClientAttributes (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/build/src/utils.js:11:65)
    at Commander.patchedConnect (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/build/src/instrumentation.js:218:68)
    at main (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/use-redis.js:29:18)
    at /Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/use-redis.js:38:11
    at AsyncLocalStorage.run (node:async_hooks:338:14)
    at AsyncLocalStorageContextManager.with (/Users/trentm/tm/opentelemetry-js-contrib11/node_modules/@opentelemetry/context-async-hooks/build/src/AsyncLocalStorageContextManager.js:33:40)
    at ContextAPI.with (/Users/trentm/tm/opentelemetry-js-contrib11/node_modules/@opentelemetry/api/build/src/api/context.js:60:46) {
  input: '',
  code: 'ERR_INVALID_URL'
}
Accessing resource attributes before async attributes settled
@opentelemetry/instrumentation-redis-4 failed to sanitize redis connection url TypeError [ERR_INVALID_URL]: Invalid URL
    at new NodeError (node:internal/errors:405:5)
    at new URL (node:internal/url:676:13)
    at removeCredentialsFromDBConnectionStringAttribute (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/build/src/utils.js:27:19)
    at getClientAttributes (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/build/src/utils.js:11:65)
    at RedisInstrumentation._traceClientCommand (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/build/src/instrumentation.js:252:60)
    at config.executor (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/build/src/instrumentation.js:151:35)
    at BaseClass.<computed> [as set] (/Users/trentm/tm/opentelemetry-js-contrib11/node_modules/@redis/client/dist/lib/commander.js:8:29)
    at main (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/use-redis.js:30:18)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/use-redis.js:38:5 {
  input: '',
  code: 'ERR_INVALID_URL'
}
@opentelemetry/instrumentation-redis-4 failed to sanitize redis connection url TypeError [ERR_INVALID_URL]: Invalid URL
    at new NodeError (node:internal/errors:405:5)
    at new URL (node:internal/url:676:13)
    at removeCredentialsFromDBConnectionStringAttribute (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/build/src/utils.js:27:19)
    at getClientAttributes (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/build/src/utils.js:11:65)
    at RedisInstrumentation._traceClientCommand (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/build/src/instrumentation.js:252:60)
    at config.executor (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/build/src/instrumentation.js:151:35)
    at BaseClass.<computed> [as get] (/Users/trentm/tm/opentelemetry-js-contrib11/node_modules/@redis/client/dist/lib/commander.js:8:29)
    at main (/Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/use-redis.js:31:30)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/trentm/tm/opentelemetry-js-contrib11/plugins/node/opentelemetry-instrumentation-redis-4/use-redis.js:38:5 {
  input: '',
  code: 'ERR_INVALID_URL'
}
GET bar: baz

I'll do a follow-up PR that uses the same guard to avoid getting the error reporting at all for this case.

trentm added a commit to trentm/opentelemetry-js-contrib that referenced this issue Aug 22, 2024
…lient URL is the empty string

Issue open-telemetry#2389 showed that this instrumentation would *crash* on a Redis
client configured with `{url: ''}` (an empty string). The crash was
fixed in open-telemetry#2397. This change avoids the once-crashy code, and hence
the diag.error spam, by using the same guard before attempting to parse
the configured client URL that the Redis client itself does, `if
(options?.url)`,

Refs: open-telemetry#2389
@rowanmanning
Copy link
Author

Thanks @trentm for the thorough explanation and quick resolution!

pichlermarc pushed a commit that referenced this issue Sep 4, 2024
…lient URL is the empty string (#2399)

Issue #2389 showed that this instrumentation would crash on a Redis
client configured with {url: ''} (an empty string). The crash was
fixed in #2397. This change avoids the once-crashy code, and hence
the diag.error spam, by using the same guard before attempting to parse
the configured client URL that the Redis client itself does, if (options?.url),

Refs: #2389
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working pkg:instrumentation-redis-4 priority:p1 Bugs which cause problems in end-user applications such as crashes, data inconsistencies
Projects
None yet
6 participants