Adding new Dell EMC ECS Object Storage Plugin for CloudStack

[mhkadhum]

Description

This PR adds support for a Dell EMC ECS S3 object storage plugin for Apache CloudStack. ECS is a software-defined object storage platform that supports both object and file-system protocols, with a focus on scalable and reliable object storage. We have been using ECS in production for three years, and this work extends CloudStack’s existing MinIO plugin to provide similar functionality for ECS.

The implementation supports the full lifecycle of S3-compatible buckets on ECS, including user provisioning, bucket creation, policy management, versioning, creation-time encryption, and integration with CloudStack’s S3 Browser. All functionality has been tested in a lab environment on Ubuntu using a CloudStack development setup based on the official installation guidelines.

Key architectural differences from MinIO:

Management API Integration
ECS requires use of the ECS Management API (port 4443, or 443 when fronted by HAProxy). CloudStack authenticates with management-user credentials to perform bucket and user operations.

Namespace Requirements
A dedicated ECS namespace is required for CloudStack-managed buckets. Multiple namespaces allow different CloudStack environments to share the same ECS cluster.

S3 Endpoints (Public and Private)
ECS exposes S3 services on ports 9020/9021 (non-TLS/TLS). In our deployment, these are routed through HAProxy and exposed externally on port 443. The Public URL is displayed to CloudStack users, while the Private URL is used internally.

TLS Handling
The “Allow Insecure HTTPS” option controls whether CloudStack accepts untrusted certificates when communicating with the ECS Management API.

User Provisioning Workflow
When a CloudStack user creates their first bucket, CloudStack provisions a corresponding ECS object user using the CloudStack UUID with a cs- prefix. ECS generates access and secret keys once, which CloudStack securely stores and reuses for subsequent bucket operations.

Bucket features:

• Encryption: Supported only during bucket creation; CloudStack hides encryption in the update view.
• Bucket Policy: Supports Private and Public configurations.
• Versioning: Fully supported through the S3 API rather than the Management API.
• Object Lock: Not supported in this release; the UI hides the option and API calls return a clear error.

Bucket modification supports quota changes, versioning updates, and policy changes. Encryption is excluded because ECS does not allow changing it after creation.

CloudStack’s S3 Browser supports upload, download, delete, listing, and prefix filtering through the ECS S3 endpoint. ECS prevents deletion of non-empty buckets, and CloudStack surfaces these errors accordingly.

We welcome review and feedback. The development fork is available here:
https://github.com/mhkadhum/cloudstack

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

The plugin was tested in an Ubuntu-based CloudStack development environment created by cloning the CloudStack source and following the official installation instructions. Testing included:

• Management API authentication and namespace operations
• Object-user provisioning and credential handling
• Bucket creation, deletion, versioning, quota updates, and policy changes
• S3 Browser operations (upload, download, delete, listing, prefix navigation)
• Error handling for invalid configurations and unsupported Object Lock requests
• TLS and insecure-TLS scenarios
• HAProxy-fronted deployments for both Management API and S3 endpoints

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

• In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
• Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
• Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
• Be sure to read the CloudStack Coding Conventions.
  Apache CloudStack is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@cloudstack.apache.org (https://cloudstack.apache.org/mailing-lists.html)
  Slack: https://apachecloudstack.slack.com/

[jbampton]

You have failing tests for pre-commit and license checks.

Have a quick read up on pre-commit it really is a series of basic checks we run:

https://github.com/apache/cloudstack/blob/main/PRE-COMMIT.md

Also we normally add the ASF license header to all files or add an exclude

[codecov]

Codecov Report

❌ Patch coverage is 0% with 1013 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.98%. Comparing base (856d83a) to head (f867823).

Files with missing lines	Patch %	Lines
...age/datastore/driver/EcsObjectStoreDriverImpl.java	0.00%	762 Missing ⚠️
...tastore/lifecycle/EcsObjectStoreLifeCycleImpl.java	0.00%	132 Missing ⚠️
.../storage/datastore/driver/EcsMgmtTokenManager.java	0.00%	60 Missing ⚠️
...udstack/storage/datastore/driver/EcsXmlParser.java	0.00%	46 Missing ⚠️
...datastore/provider/EcsObjectStoreProviderImpl.java	0.00%	13 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12124      +/-   ##
============================================
- Coverage     18.01%   17.98%   -0.04%     
- Complexity    16607    16609       +2     
============================================
  Files          6029     6034       +5     
  Lines        542154   543210    +1056     
  Branches      66451    66667     +216     
============================================
  Hits          97681    97681              
- Misses       433457   434513    +1056     
  Partials      11016    11016              

Flag	Coverage Δ
uitests	3.52% <ø> (-0.01%)	⬇️
unittests	19.13% <0.00%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sureshanaparti]

@mhkadhum can you check the build errors?

[mhkadhum]

Thanks for the feedback!

ive added the missing license headers and ran pre-commit locally, and pushed an updated commit

[DaanHoogland]

looks generally good @mhkadhum , thanks for the contribution. Will you be maintaining this? As it is a 3rd party component and it may require hardware or licenses not available to the project.

[mhkadhum]

looks generally good @mhkadhum , thanks for the contribution. Will you be maintaining this? As it is a 3rd party component and it may require hardware or licenses not available to the project.

Yes, I will be the maintainer for the plugin. I have made a new commit to fix the previous build. Could you kindly review it

[DaanHoogland]

I have not found any oddities in the code, but one remark (in two parts ;) ) Feel free to ignore but I think it will help in the future.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[abh1sar]

@mhkadhum please fix build errors
EcsObjectStoreDriverImpl.java:120: Line has trailing spaces. [RegexpSingleline]

[mhkadhum]

@mhkadhum please fix build errors EcsObjectStoreDriverImpl.java:120: Line has trailing spaces. [RegexpSingleline]

Missed the pre-commit hook. Fixed and pushed.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 16355

[abh1sar]

@blueorangutan package

[blueorangutan]

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[mhkadhum]

Hello @DaanHoogland ,

Can u approve the pipeline ?
Thanks

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 16379

[mhkadhum]

@blueorangutan package

Hello @DaanHoogland ,
Can u test, I believe this is ready to merge.

[blueorangutan]

@mhkadhum a [SL] Jenkins job has been kicked to build packages. It will be bundled withHello @DaanHoogland ,
Can u test, I believe this is ready to merge. SystemVM template(s). I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16381

[abh1sar]

@blueorangutan test

[blueorangutan]

@abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-15191)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 55310 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12124-t15191-kvm-ol8.zip
Smoke tests completed. 148 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_scale_kubernetes_cluster	Failure	42.19	test_kubernetes_clusters.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	Failure	293.88	test_vpc_redundant.py

[winterhazel]

@mhkadhum thanks for the pull request, and sorry for the lack of attention to it recently. As @abh1sar already validated that there were no regression for other plugins in #12124 (comment), I will merge it to 4.23 with one more approval. The test failures in #12124 (comment) are unrelated.

I intend to review it this week.

Also, as you will be maintaining this plugin, you can add a line to .github/CODEOWNERS if you want to containing your GitHub handle so that you are automatically marked as a reviewer whenever a PR that affects the plugin is created.

[mhkadhum]

@mhkadhum thanks for the pull request, and sorry for the lack of attention to it recently. As @abh1sar already validated that there were no regression for other plugins in #12124 (comment), I will merge it to 4.23 with one more approval. The test failures in #12124 (comment) are unrelated.

I intend to review it this week.

Also, as you will be maintaining this plugin, you can add a line to .github/CODEOWNERS if you want to containing your GitHub handle so that you are automatically marked as a reviewer whenever a PR that affects the plugin is created.

Hi @winterhazel

I have synced the fork with its upstream source and registered myself within the CODEOWNERS file

[winterhazel]

@blueorangutan package

[blueorangutan]

@winterhazel a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17600