API to list console sessions

[bernardodemarco]

Description

Currently, details about console sessions are stored in the cloud.console_session table in the database. Operators can only access this information by querying the database directly, while end users have no way to view console session data at all.

To address this, this PR proposes to create the listConsoleSessions API. It allows listing the console sessions, with optional filters by domain, account, user, host, instance, IP address, and date. The API is accessible to all account types and, thus, performs proper access validation on the queried resources.

The API supports the following parameters:

Parameter	Description	Required
id	Console session ID.	No
activeonly	Indicates whether only currently active console sessions should be listed. Defaults to true. Active sessions are the ones that have been acquired and have not been removed.	No
acquired	Indicates whether only acquired console sessions should be listed. Defaults to false. Acquired sessions are the ones that have been accessed. The activeonly parameter has precedence over the acquired parameter, i.e., when the activeonly parameter is true, the acquired parameter value will be ignored.	No
isrecursive	Lists console sessions recursively per domain. Defaults to false.	No
clientaddress	IP address of the client that accessed the console.	No
consoleendpointcreatoraddress	IP address of the client that generated the console session endpoint.	No
hostid	ID of the host where the VM was running when the console session endpoint was generated. Only available for Root Admins.	No
instanceid	ID of the VM associated with the console session.	No
startdate	If provided, only console sessions that were generated from this date onward will be listed.	No
enddate	If provided, only console sessions that were generated up to this date will be listed.	No
domainid	Domain ID of the account that generated the console session endpoint.	No
accountid	ID of the account that generated the console session endpoint.	No
userid	ID of the user that generated the console session endpoint.	No
page	Used for pagination along with pagesize.	No
pagesize	User for pagination along with page.	No

This PR only encompasses the creation of the API listConsoleSessions API. UI support will be implemented in a future PR.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Screenshots (if appropriate):

How Has This Been Tested?

Tests Setup

1. Created the following domain hierarchy:

ROOT
├── d1
│   └── d1-d1
└── d2

2. Created the following accounts:

Name	Role Type	Domain
admin	Admin	ROOT
u1	User	ROOT
d1-admin	DomainAdmin	ROOT/d1
d1-user	User	ROOT/d1
d1-d1-admin	DomainAdmin	ROOT/d1/d1-d1
d1-d1-user	User	ROOT/d1/d1-d1
d2-admin	DomainAdmin	ROOT/d2

3. Deployed a VM for each account:

ID	Name	Instance Name	Account Name
5	v-5-VM	v-5-VM	system
6	s-6-VM	s-6-VM	system
7	VM-fb5fa7ab-c5a0-4943-88bd-dedec9007c7b	i-7-7-VM	d1-d1-user
8	r-8-VM	r-8-VM	d1-d1-user
9	VM-16febbcd-a35e-4f05-87d3-1a27db077136	i-6-9-VM	d1-user
10	r-10-VM	r-10-VM	d1-user
11	VM-d3208889-d3fd-43fe-9087-3cf5d863a3b1	i-4-11-VM	d1-admin
12	r-12-VM	r-12-VM	d1-admin
13	VM-dea9f800-afd0-4054-9160-3a676a0df3d3	i-2-13-VM	admin
14	r-14-VM	r-14-VM	admin
15	VM-766b6485-330e-4ec9-9516-59e17b52c1c7	i-8-15-VM	u1
16	r-16-VM	r-16-VM	u1
17	VM-d2d7ded1-e210-4191-9a0e-75c3914de043	i-9-17-VM	d2-admin
18	r-18-VM	r-18-VM	d2-admin
19	VM-266b73ff-2684-46b9-b36e-830ac83c5c14	i-5-19-VM	d1-d1-admin
20	r-20-VM	r-20-VM	d1-d1-admin

4. With each account, I accessed all available VM consoles through the CPVM.

listConsoleSessions tests

• With the admin account, verified that the API lists console sessions correctly according to the specified parameters
• With the u1 account, verified that it is only possible to access the account's console sessions
• With the d2-admin account, verified that it is only possible to list the console sessions of the d2 domain
• With the d1-admin account, verified that it is only possible to list the console sessions of the d1 and d1/d1-d1 domains
• With the d1-user account, verified that it is only possible to list the d1-user console sessions
• With the d1-d1-admin account, verified that it is only possible to list the console sessions of the d1/d1-d1 domain
• With the d1-d1-user account, verified that it is only possible to list the d1-d1-user console sessions
• With accounts of the User type, verified that the hostid parameter is not considered in the API workflow
• With accounts of the User type, verified that the hostid and hostname response attributes are not included in the API's return

[bernardodemarco]

@blueorangutan package

[codecov]

Codecov Report

Attention: Patch coverage is 56.36943% with 137 lines in your changes missing coverage. Please review.

Project coverage is 16.60%. Comparing base (0bd35a5) to head (475460e).
Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
...loudstack/api/response/ConsoleSessionResponse.java	50.48%	51 Missing ⚠️
...n/java/com/cloud/vm/dao/ConsoleSessionDaoImpl.java	9.09%	40 Missing ⚠️
...mand/user/consoleproxy/ListConsoleSessionsCmd.java	50.81%	30 Missing ⚠️
...a/src/main/java/com/cloud/vm/ConsoleSessionVO.java	14.28%	6 Missing ⚠️
...udstack/consoleproxy/ConsoleAccessManagerImpl.java	87.50%	5 Missing and 1 partial ⚠️
...src/main/java/com/cloud/api/ApiResponseHelper.java	94.00%	1 Missing and 2 partials ⚠️
...in/java/com/cloud/server/ManagementServerImpl.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11016      +/-   ##
============================================
+ Coverage     16.57%   16.60%   +0.03%     
- Complexity    13968    14020      +52     
============================================
  Files          5743     5745       +2     
  Lines        510470   511012     +542     
  Branches      62074    62138      +64     
============================================
+ Hits          84615    84868     +253     
- Misses       416393   416668     +275     
- Partials       9462     9476      +14     

Flag	Coverage Δ
uitests	3.90% <ø> (-0.01%)	⬇️
unittests	17.50% <56.36%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bernardodemarco]

@blueorangutan package

[blueorangutan]

@bernardodemarco a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13755

[bernardodemarco]

@blueorangutan package

[blueorangutan]

@bernardodemarco a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[DaanHoogland]

clgtm, only doubt, is the active parameter needed? would we ever want to list no longer available sessions?

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13766

[blueorangutan]

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 13771

[bernardodemarco]

@DaanHoogland, thanks for the review!

is the active parameter needed? would we ever want to list no longer available sessions?

Yes, listing removed sessions is useful for audit and analysis purposes, as it allows users to track who generated a console endpoint, who accessed it, and when it was generated, acquired, and removed. Listing only active sessions, on the other hand, is helpful for verifying whether someone is currently using a VM.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13777

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-13527)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 54933 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11016-t13527-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[DaanHoogland]

@blueorangutan test ol8 vmware-70u3 keepEnv

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-13630)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 58453 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11016-t13630-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[bernardodemarco]

@blueorangutan package

[blueorangutan]

@bernardodemarco a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[bernardodemarco]

I have only one minor request: instance_id is not autocompleting on cmk, can this autocompletion be added?

@nvazquez, I've just created the apache/cloudstack-cloudmonkey#162 PR to add autocomplete support for the instanceid parameter.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13986

[DaanHoogland]

@blueorangutan test keepEnv

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-13663)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 58048 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11016-t13663-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_enableHumanReadableLogs	Error	0.26	test_human_readable_logs.py

[bernardodemarco]

@JoaoJandre, thanks for testing!

I have just applied these suggestions #11016 (comment).

Now, the startdate and enddate API parameters are compared to the created attribute of console sessions, see 90b09ea. Additionally, the acquired parameter has been introduced, see 475460e. It is responsible for listing only acquired sessions, that is, sessions that have been accessed. The activeonly parameter has precedence over the acquired parameter. Thus, when activeonly is true, the value of acquired is ignored.

Therefore:

• when executing list consolesessions isrecursive=true, all active console sessions (acquired and not removed) are listed;
• when executing list consolesessions isrecursive=true activeonly=false, all console sessions are listed;
• when executing list consolesessions isrecursive=true activeonly=false acquired=true, all acquired console sessions (only marked as acquired in the DB) are listed; and,
• when executing list consolesessions isrecursive=true activeonly=true acquired=true, all active console sessions are listed.

[bernardodemarco]

@blueorangutan package

[blueorangutan]

@bernardodemarco a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14041

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-13691)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 53149 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11016-t13691-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[JoaoJandre]

LGTM. Did some basic tests mixing and matching the fields of the API and all seemed ok.

[JoaoJandre]

@nvazquez do you have any concerns regarding this PR? Or may we merge it?

[nvazquez]

@bernardodemarco @JoaoJandre sorry I was on vacation last week, I'll post my review soon

[nvazquez]

Thanks @bernardodemarco I have updated my env to the latest packages and used your cmk PR with a slight modification of the instanceid parameter for the autocompletion listing console sessions for a Domain admin. Overall looks good!

I noticed after upgrading my env to the latest packages there was a session still being listed which was not active but still part of the response in the API. I managed to reproduce it by opening a new session for a VM, leaving the session open on an inactive browser tab until the tab title switches from the VM name to 'noVNC'. After that, the session for that VM keeps being listed even with activeonly=true despite closing the tab. Have you faced this case before? I don't think it is an issue with this PR but perhaps there could be a mechanism to remove this kind of 'incorrect' console sessions from the API response

[nvazquez]

Hi @bernardodemarco there is a mismatch between this parameter value and the autocompletion cmk PR name (instance_id vs instanceid). For simplicity, can this value be changed to instanceid?