Collect more historical data from cloud-based brokers#5522
Collect more historical data from cloud-based brokers#5522mikeminutillo wants to merge 1 commit into
Conversation
AWS 365 days Azure 90 days Matches Particular/Particular.EndpointThroughputCounter#1086
| : endpoint.LastCollectedDate.AddDays(1); | ||
|
|
||
| await foreach (var queueThroughput in brokerThroughputQuery.GetThroughputPerDay(queueName, startDate, stoppingToken)) | ||
| await foreach (var queueThroughput in brokerThroughputQuery.GetThroughputPerDay(queueName, endpoint.LastCollectedDate.AddDays(1), stoppingToken)) |
There was a problem hiding this comment.
a side issue, but can we validate the .AddDays(1)? I remember looking at this a while back and working out that every time the ServiceControl service is restarted for any reason, we lose a day of data collection for some transports
There was a problem hiding this comment.
That would happen for RabbitMQ and SQL, which do not use this date at all. Their broker queries start when they are called and run for 24 hours before returning. If they stop mid-collection, all of the collected data is lost and the timer starts again.
For Azure and SQS (which do use this date), I believe the behavior is correct. Query for the last date we have data for and then start collection from the next day.
There was a problem hiding this comment.
that would still mean that if, on ASB or SQS, someone closed ServiceControl at 1AM on 1/1 and started it again 2 days later, we would lose 23 hours of data from 1/1 and only start again from 2/1
There was a problem hiding this comment.
It shouldn't. The behavior of Azure and SQS is to collect data from the day after the last recorded date up to yesterday. We never get partial days of data. So in the example you gave, we would not have recorded anything for the day when the service stops at 1am. But when you restart 2 days later we'll query for that day and the day after.
| { | ||
| startDate = earliestQueryDate; | ||
| } | ||
| var endDate = today.AddDays(-1); // Query date up to but not including today |
There was a problem hiding this comment.
this comment exists here but not on ASB, however it doesn't explain why
2 participants
Matches a similar change made in the endpoint throughput tool.