camel-aws-xray tracing support

[RovoMe]

As requested by @oscerd I open the PR for the master-branch this time. I copy over the other information from the original 2.19.x branch PR #2071

---

Added support for AWS XRay tracing (similar to OpenTracing) which tracks the lifecycle of the exchange and invoked EIPs and bean/processors and adds the gathered information to AWS XRay segments. The code itself is currently added to the 2.19.x branch as we still use 2.19.1 internally.

AWS XRay will send the data to a locally running daemon listening on UDP port 2000 which sends the gathered data to AWS and presents it as visualization as for service maps and actual traces

In the trace image above, a segment corresponds to the bold entries, like api-file-upload and process-file while anything indented is a subsegment of the corresponding segment.

The code itself does create a new AWS XRay segment for a new exchange unless there is already an active segment available within the threads context. AWS XRay manages its segments on a thread-local basis and discards any information gathered in case a new segment is spawned although an already active segment is present in the threads context. To circumvent data loss, the implementation will check first if already an active segment is present in the current threads context and only then create a new one. If already an active segment is present (in case of .to("direct:foo") the creation of a new exchange will add a subsegment to the current segment. As multicast with parallel processing enabled will spawn new threads in a separate thread, the segment is copied over via an exchange property and set manually for threads which name contains a multicast name.

The images above contains only traces of .bean(...) and .process(...) instances which are a annotated with @XRayTrace. This annotation is processed by TraceAnnotatedTracingStrategy which adds subsegments for the respective beans to the AWS XRay segment. If no tracing strategy (= InterceptStrategy) is configured, this implementation will default to a NoopTracingStrategy which as its name suggest will do nothing. A very simple EIPTracingStrategy is also provided which adds subsegments for the currently processed ProcessorDefinitions short name to the current segment.

[oscerd]

You need to run also the -Psourcecheck to verify the code style

[oscerd]

All the version should be 2.21.0-SNAPSHOT in this PR.

[oscerd]

@davsclaus can you give a look?

[davsclaus]

Oh there was another PR - I commented there. They same applies for this one.

[oscerd]

@davsclaus I'll wait for you to start the merging process.

[RovoMe]

I'm not sure if this PR is really ready for merging yet. I've not yet tested the segment.putSql(key, value) and hence I'm not sure how this differs from segment.putMetadata(key, value) nor if it supports elasticsearch or mongodb queries, which I currently have added as well. In general, AWS XRay allows to add metadata or annotations to a (sub)segment. The difference between both is, that the latter is indexed and thus searchable while the former one is just attached, though not (yet) directly visible via their interface.

A thing that bothers me still somehow is, that the decorator classes added just append selected information for certain endpoints as metadata (as of now) to the given (sub)segment. The trace generated though does not contain route-interna like choices or other processor definitions by default as they are not captured by the RoutePolicySupport or by the EventNotifierSupport implementation. Hence I kept the basic EIPTracingStrategy for now as some application might want to track certain features like which branching decision was performed (in a choice definition) or similar stuff. The company I'm working for i.e. will use TraceAnnotatedTracingStrategy in order to monitor which beans and processors got invoked throughout the lifecycle of the exchange and might want to add branching decision later on if needed.

[oscerd]

@davsclaus did you get a chance to review the requested changes?

[davsclaus]

@oscerd not yet - will try to find some time tomorrow. There is a bit of comments as well to read as well

[RovoMe]

Just to give a quick headsup on the current status. The images below are taken from my test project which defines a custom intercept strategy in order to learn which processor definitions got invoked. Note that this sample project does not aim to solve any real world issues and is therefore just a tech-demo.

The exchange creation/closing is handled by a RoutePolicySupport in XRayTracer while any forwards of the exchange are handled by an EventNotifierSupport implementation. However, as these only trigger when an exchange is created, copied (~ passed further) or removed this will actually not handle any internals such as using .bean(...), .enrich(...) or .log(...) while .to("log:...") or .to("bean:...") will be handled by the above mentioned implementations. Similar to OpenTracing, this project does now also contain decorators as well which kick in on i.e. .to(...) definitions and add additional metadata to the segment as can be seen hopefully in the image below which contains the actual invoked URI in the metadata tab of the processed camel-https component (https:www.google.at sub-segment in the picture above)

The first image also indicates that an error occurred during the SQL query. This error is also set by the SqlSegmentDecorator in the post-phase. On hovering over the exclamation mark the reason for the error is also visible:

On clicking on the exclamation mark AWS will present the stacktrace that lead the component to fail:

While I think that this project is on a good track, I don't think it is ready yet for production and needs some more finetuning. I will try to get MySQL, MongoDB and eventually Elasticsearch queries up and running in my sample project in order to determine if AWS can handle other queries besides SQL as well.

[RovoMe]

As I managed to get the SQL sample route to work in my tech demo, I experimented with the decorator a bit and, well, the putSql method does different things than I expected it to do. I looked up what AWS XRay does on adding the xray-mysql dependency to the project, which unfortunately requires a Tomcat based JdbcInterceptor pool. The SqlSegmentDecorator would need to change to something like this (not yet commited to the PR branch) to behave similar to AWS Ray's (My)SQL tracer though this would require to add camel-sql as dependency to the project or move this decorator to an own project like camel-aws-xray-mysql instead and add a respective ServiceLoader configuration as well to it.

With the above mentioned changes AWS XRay will present the actual query issued in the meta data of the trace

while the additional metadata gathered by the decorator are added via a separate SQL tab

This tab will filter any unknown properties. While it would allow to add an additional preparation element that can either be call for a CallableStatement or statement for a PreparedStatement, adding a custom query entry with the plain text search query won't appear in the actual SQL tab though.

As pulling in additional information into the trace might require to add further dependencies, what is your prefered strategy here? Add the dependencies to camel-aws-xray? Move the decorators to sub-projects like camel-aws-xray-sql or leave that kind of data collection to the user itself who needs that kind of information and therefore just implement a very minimal decorator?

[oscerd]

I guess this PR will need a bit of time to get merged. We'll wait for @davsclaus feedback, since he's the one that worked more on tracing stuff.

[oscerd]

@davsclaus, maybe we can merge this for the moment and do some changes in the future. WDYT?

[oscerd]

Do you know if opentracing or Zipkin can be used to bridge to AWS Xray? Because in that case, we can use them instead of this custom component.

The intercept wrapper doesn't not work with async non blocking routing engine. This is a concern about this component.

Also, we need to test this against AWS.. Btw I think all your comments, images and stuff can be added to the .adoc, we'll review soon.

[RovoMe]

There may be some implementations which I am not aware of that probably bridge OpenTracing spans to XRay (sub)segments. I've only found this python project and some thoughts to position OpenTracing as vendor-neutral bridge for other tracing frameworks so far.

I haven't used Zipkin nor OpenTracing so far and thus can't state if the XRay support can be handled via one of the other Camel components. OpenTracing seems to work with spans while XRay uses segments and subsegments. I'm not sure if OpenTracing also utilizes a thread-local acting version of AWSRay in order to manage the lifecycle of spans. If so, it might be possible to unify the approach somehow in order to support both, OpenTracing and XRay.

The intercept wrapper doesn't not work with async non blocking routing engine. This is a concern about this component.

The RoutePolicySupport and the EventNotifierSupport both don't trace invoked ProcessorDefinitions hence I added the InterceptStrategy. Please advice on how to trace those invoked beans, processors, ... instead :)

Btw I think all your comments, images and stuff can be added to the .adoc

The .adoc documentation needs some update as the code changed notably since I wrote that information. I will see if I can find some time at weekends to update it

[oscerd]

We merged the PR, any other update or something (docs, fixes etc) can be in other PRs.

Thanks :-)

[rmaugusto]

I suggest add some additional information to AbstractSegmentDecorator to extract Rest endpoint data.
Just like AWSXRayServletFilter.

Also, it would be nice to allow the inclusion of custom decorators.

[davsclaus]

@rmaugusto you are surely welcome to contribute. You can create a JIRA ticket for this enhancement
https://camel.apache.org/manual/latest/contributing.html