Skip to content

Batch NIC and VLAN reads in collectVmNetworkStatistics to reduce per-NIC DB round-trips#13474

Draft
vishesh92 wants to merge 2 commits into
apache:4.22from
shapeblue:optimize-network-stats-batch-reads
Draft

Batch NIC and VLAN reads in collectVmNetworkStatistics to reduce per-NIC DB round-trips#13474
vishesh92 wants to merge 2 commits into
apache:4.22from
shapeblue:optimize-network-stats-batch-reads

Conversation

@vishesh92

@vishesh92 vishesh92 commented Jun 23, 2026

Copy link
Copy Markdown
Member

Description

collectVmNetworkStatistics performed per-NIC: search by MAC, listVlansByNetworkId, producing 2 * NIC_COUNT queries per VM. Now batch-loads all NICs by MAC addresses and all VLANs by network IDs upfront (2 queries total for the read side).

Add NicDao.listByMacAddresses(List) for batch MAC lookup
Add VlanDao.listVlansByNetworkIds(List) for batch network VLAN lookup
Replace per-iteration NIC search + VLAN lookup with map lookups
Write side (lock + update) intentionally unchanged — row-level lock is load-bearing

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@vishesh92

vishesh92 commented Jun 23, 2026

Copy link
Copy Markdown
Member Author

@blueorangutan package

@blueorangutan

blueorangutan commented Jun 23, 2026

Copy link
Copy Markdown

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@codecov

codecov Bot commented Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 2.70270% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.66%. Comparing base (288f9a9) to head (03e3ab2).

Files with missing lines Patch % Lines
.../src/main/java/com/cloud/vm/UserVmManagerImpl.java 0.00% 21 Missing ⚠️
...ema/src/main/java/com/cloud/vm/dao/NicDaoImpl.java 0.00% 8 Missing ⚠️
...ma/src/main/java/com/cloud/dc/dao/VlanDaoImpl.java 12.50% 7 Missing ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               4.22   #13474      +/-   ##
============================================
- Coverage     17.67%   17.66%   -0.01%     
- Complexity    15790    15791       +1     
============================================
  Files          5922     5922              
  Lines        533173   533204      +31     
  Branches      65209    65214       +5     
============================================
- Hits          94218    94206      -12     
- Misses       428309   428353      +44     
+ Partials      10646    10645       -1
Flag Coverage Δ
uitests 3.69% <ø> (ø)
unittests 18.74% <2.70%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blueorangutan

blueorangutan commented Jun 23, 2026

Copy link
Copy Markdown

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 18343

sliceofapplepie and others added 2 commits June 23, 2026 18:58
@sliceofapplepie @vishesh92
Batch NIC and VLAN reads in collectVmNetworkStatistics to reduce per-…
a5dd545 
…NIC DB round-trips

collectVmNetworkStatistics performed per-NIC: search by MAC, listVlansByNetworkId,
producing 2 * NIC_COUNT queries per VM. Now batch-loads all NICs by MAC addresses
and all VLANs by network IDs upfront (2 queries total for the read side).

- Add NicDao.listByMacAddresses(List<String>) for batch MAC lookup
- Add VlanDao.listVlansByNetworkIds(List<Long>) for batch network VLAN lookup
- Replace per-iteration NIC search + VLAN lookup with map lookups
- Write side (lock + update) intentionally unchanged — row-level lock is load-bearing
@vishesh92
Reuse cached search builders with IN operator and guard empty lists i…
03e3ab2 
…n batch NIC/VLAN reads
@vishesh92 vishesh92 force-pushed the optimize-network-stats-batch-reads branch from 6c3052f to 03e3ab2 Compare June 23, 2026 13:29
@vladimirpetrov

vladimirpetrov commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

@blueorangutan package

@blueorangutan

blueorangutan commented Jun 25, 2026

Copy link
Copy Markdown

@vladimirpetrov a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

blueorangutan commented Jun 25, 2026

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18375

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@vishesh92 @blueorangutan @vladimirpetrov @sliceofapplepie