[Playground] OTLP-Mapper - instana to otlp span mapper

[abhilash-sivan]

Summary

Introduces the Instana-to-OTLP converter playground.

• Adds support for exporting traces via OTLP (4318/v1/traces).
• Supports Semantic Conventions (SemConv) v1.23.

Not Included

• Metrics export
• Logs export

TODO

• Handle Agent port usage , data vs connection port chore: make agent data port configurable independently from the agent connection port #2559
• Handle multiple data keys eg: span.data.peer, span.data.mongo etc
• Overall design
• Implement SemConv lookup handling.
• Implement span name mapping.
• Complete metadata attribute mapping:
• Direct mappings
• Computed mappings
• Reserach span status setting
• Research span name maping @abhilash-sivan
• OpenTelemetry instr specific handling
• Log-specific handling
• Resource mapping
• Implement required metrics mappings.

• Complete span data mapping:

• HTTP
• Kafka
• Databases
• Cloud services
• OpenTelemetry instrumentation

• Add test coverage:

• Single-span scenarios
• Multi-span scenarios
• Perform E2E validation:

e2e Testing @abhilash-sivan

• SemConv v1.23 with the current design
• SemConv v1.41 compatibility validation
• OpenTelemetry wrapper instrumentation testing
• metadata in v1.23 and span data in v1.43

results note: https://ibm.box.com/s/0v5wli0pgvrlg8jqpkhi59mb52o28u75

Special cases

• Fix types

Validation

Before merging, verify:

• Traces are successfully exported through the OTLP endpoint (4318/v1/traces).
• Metrics continue to be exported through the standard endpoint.
• The UI correctly identifies and displays applications.
• The configuration option to enable the OTLP format, INSTANA_OTLP_ENABLED or?

Future Enhancements

• Support the agent announcement cycle disable use case.
• Evaluate exporting logs to the OTLP /v1/logs endpoint.
• Metrics exporting

Refrences

https://opentelemetry.io/blog/2023/http-conventions-declared-stable/

[abhilash-sivan]

This is where the conversion is happening.

This seems to be the optimal place because if we convert early, then in some places we talk Instana format and it could raise unexpected issues. So before transmission seems simple and maintanable

[aryamohanan]

Should we do this here or just after the current BE transformation,
https://github.com/instana/nodejs/pull/2534/changes?utm_source=chatgpt.com#diff-462ace70d7ac6eb4cd6f82756059c88629bfad1463fde0498d89106d7ccc8104L193

Right now we’re applying the transformation after collecting all spans. IMO it would be cleaner to do it before calling transmitSpans, so the transmit layer only deals with already-transformed spans.

TBD

[abhilash-sivan]

Yeah, I agree. This would be a more cleaner way

// Transform internal span data format into external (backend) readable format.
span = applySpanTransformation(span);

Just after this transformation, we can call the Otel transformation.

[abhilash-sivan]

One thing we have to consider is that currently all the spans intended for transmission — i.e., the array of spans — are converted together. If we do it in the above-mentioned area, we will have to process the spans individually.

[abhilash-sivan]

Note: TBD — Transform each span to OTLP format immediately in addSpan and store them in OTLP format in the buffer if batching is enabled. This would require changes to the batching logic.

[kirrg001]

How do you want to organize the agent ports etc?

OTLP mode is on -> port will change.
There is something missing.

[kirrg001]

Also:
How can we skip the agent announcement cycle when connecting to an otel collector later?
Did you draw that path as well to confirm the architecture?

[abhilash-sivan]

No, I haven’t worked much on the architecture yet. I can work on the dynamic port connections part, and I’ll also think through the agent announcement cycle and how we can skip it when connecting to an OTel collector

[kirrg001]

test: send traces via otlp port and metrics via standard port - Idenfies 2 diff apps 1. Otel and 2. Node.js

This needs to be communicated in the next WG. Traces & Metrics mapping is required.

[kirrg001]

I'd suggest to load the mappings once before the steps start and provide the mappings via dependency injection to the generic modules.

const mappings = getMappings()

// steps
// meta(mappings)
// etc

[kirrg001]

This is not needed.
If you know the type e.g. pg (instana).
You know the otlp target mapping and you know its in db group.

[aryamohanan]

This is the future design in my mind now

└── converters/                       
    └── otlp/                        
        ├── index.js                  ← Entry point (exports transform())
        ├── converter.js              ← Core conversion logic
        ├── constants.js              ← Metric types, aggregation types
        ├── util.js                   ← Helper functions
        ├── transformers/             
        │   ├── index.js
        │   ├── resourceAttributes.js ← Service name, SDK info, host, PID
        │   └── metricData.js         ← Metric value transformation
        └── mappers/                  
            └── metricAttributes.js   

Share your thoughts 💭