Skip to content

Collect more historical data from cloud-based brokers #5522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mikeminutillo wants to merge 1 commit into
base: master
Choose a base branch
from
+18 −9
Open

Collect more historical data from cloud-based brokers #5522

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions src/Particular.LicensingComponent/BrokerThroughput/BrokerThroughputCollectorHostedService.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,12 +98,7 @@ async Task Exec(IBrokerQueue queueName, string postfix)
await dataStore.SaveEndpoint(endpoint, stoppingToken);
}

var defaultStartDate = DateOnly.FromDateTime(timeProvider.GetUtcNow().DateTime).AddDays(-30);
var startDate = endpoint.LastCollectedDate < defaultStartDate
? defaultStartDate
: endpoint.LastCollectedDate.AddDays(1);

await foreach (var queueThroughput in brokerThroughputQuery.GetThroughputPerDay(queueName, startDate, stoppingToken))
await foreach (var queueThroughput in brokerThroughputQuery.GetThroughputPerDay(queueName, endpoint.LastCollectedDate.AddDays(1), stoppingToken))

@PhilBastian PhilBastian Jun 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a side issue, but can we validate the .AddDays(1)? I remember looking at this a while back and working out that every time the ServiceControl service is restarted for any reason, we lose a day of data collection for some transports

@mikeminutillo mikeminutillo Jun 9, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would happen for RabbitMQ and SQL, which do not use this date at all. Their broker queries start when they are called and run for 24 hours before returning. If they stop mid-collection, all of the collected data is lost and the timer starts again.

For Azure and SQS (which do use this date), I believe the behavior is correct. Query for the last date we have data for and then start collection from the next day.

@PhilBastian PhilBastian Jun 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that would still mean that if, on ASB or SQS, someone closed ServiceControl at 1AM on 1/1 and started it again 2 days later, we would lose 23 hours of data from 1/1 and only start again from 2/1

@mikeminutillo mikeminutillo Jun 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shouldn't. The behavior of Azure and SQS is to collect data from the day after the last recorded date up to yesterday. We never get partial days of data. So in the example you gave, we would not have recorded anything for the day when the service stops at 1am. But when you restart 2 days later we'll query for that day and the day after.

{
try
{
Expand Down
10 changes: 8 additions & 2 deletions src/ServiceControl.Transports.ASBS/AzureQuery.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ public class AzureQuery(ILogger<AzureQuery> logger, TimeProvider timeProvider, T
{
const string CompleteMessageMetricName = "CompleteMessage";
const string MicrosoftServicebusNamespacesMetricsNamespace = "Microsoft.ServiceBus/Namespaces";
const int MaxDaysToQuery = 90;

string serviceBusName = string.Empty;
ArmClient? armClient;
Expand Down Expand Up @@ -202,8 +203,13 @@ public override async IAsyncEnumerable<QueueThroughput> GetThroughputPerDay(IBro
[EnumeratorCancellation] CancellationToken cancellationToken)
{
logger.LogInformation($"Gathering metrics for \"{brokerQueue.QueueName}\" queue");

var endDate = DateOnly.FromDateTime(timeProvider.GetUtcNow().DateTime).AddDays(-1);
var today = DateOnly.FromDateTime(timeProvider.GetUtcNow().DateTime);
var earliestStartDate = today.AddDays(-MaxDaysToQuery);
if (startDate < earliestStartDate)
{
startDate = earliestStartDate;
}
var endDate = today.AddDays(-1);
if (endDate < startDate)
{
yield break;
Expand Down
10 changes: 9 additions & 1 deletion src/ServiceControl.Transports.SQS/AmazonSQSQuery.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ namespace ServiceControl.Transports.SQS;
public class AmazonSQSQuery(ILogger<AmazonSQSQuery> logger, TimeProvider timeProvider, TransportSettings transportSettings)
: BrokerThroughputQuery(logger, "AmazonSQS")
{
const int MaxDaysToQuery = 365;

AmazonCloudWatchClient? cloudWatch;
AmazonSQSClient? sqs;
string? prefix;
Expand Down Expand Up @@ -198,7 +200,13 @@ public override async IAsyncEnumerable<QueueThroughput> GetThroughputPerDay(IBro
[EnumeratorCancellation] CancellationToken cancellationToken)
{
var utcNow = timeProvider.GetUtcNow();
var endDate = DateOnly.FromDateTime(utcNow.DateTime).AddDays(-1); // Query date up to but not including today
var today = DateOnly.FromDateTime(utcNow.DateTime);
var earliestQueryDate = today.AddDays(-MaxDaysToQuery);
if (startDate < earliestQueryDate)
{
startDate = earliestQueryDate;
}
var endDate = today.AddDays(-1); // Query date up to but not including today

@PhilBastian PhilBastian Jun 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this comment exists here but not on ASB, however it doesn't explain why


var isBeforeStartDate = endDate < startDate;

Expand Down
Loading