Add third party plugins #54

Mark-Powers · 2021-11-30T20:27:17Z

This adds third party plugins, by creating a robust base class that manages the API, the service functions, the policies, and monitoring.

Notes for review:

the Host third party plugin was created to "show off" what this is capable of. Less important for us, but more of interest if this code is eventually merged upstream.
the base third party plugin is where most of the code is, essentially all of the plugin boilerplate.
The third party plugins all share the same db models, and custom data on the resource is stored in a JSON column.

…oud/blazar into third_party_plugins

…tus to ERROR" This reverts commit d78c38a. Change-Id: Ia80e00a1aac3ad0c02a5062e997447fe243bacc6

Change-Id: Iaa8a88fb96d32c0bfe351af6115550bef5f80889

Change-Id: I8263254dc7710d57bd50cd37ea3da2485dbbb38a

Change-Id: Ib1a06802ba827b2159de4fc61c97f2df273aff36

Change-Id: Idb31b4a048d6499cab6bbe1b90b93599c519a442

Change-Id: Ic9bc1c9645aee9c7a98fd1d8ecb8f0169404161d

Change-Id: I9f0d7f47e92ed222cbb7cd042304317f6c2e74ee

Change-Id: Ia0bd2e39327b42597eae60a54a3050cc6ff0764c

Change-Id: I714c7389067e7d6d625790a3f776a07484652cb9

Mark-Powers · 2023-06-01T19:20:27Z

Note: This git history is too messy to merge on it's own, we should squash.

…ud/blazar into third_party_plugins

super-cooper

A lot of comments, but considering the sheer volume of code here, I think that's a pretty good ratio! I tried my best to fully understand everything going on here, and it helped that I've been working a little bit on plugins recently, but I did find myself getting a bit lost in the sauce. I hope my comments make sense and are useful.

super-cooper · 2023-06-15T21:01:09Z

blazar/db/sqlalchemy/api.py

+        # Add extra filter if using 3rd party resource type
+        if resource_type not in EXTRA_CAPABILITY_MODELS:
+            query = query.join(models.Resource).filter_by(
+                resource_type=resource_type)


This looks like it could be consolidated into the above else body instead of performing this check again.

super-cooper · 2023-06-15T21:54:40Z

blazar/manager/service.py

-                        "An unexpected exception type was generated. This "
-                        "indicates that some exception is not being wrapped "
-                        "properly in a BlazarException."))
+            except common_ex.BlazarException:


Is there a reason why we're excluding the warning that used to be here?

This only exists in our fork, I'll revert my change for now

super-cooper · 2023-06-15T22:02:01Z

blazar/manager/service.py

+                'status': status.reservation.PENDING
+            }
+            reservation = db_api.reservation_create(reservation_values)
+            plugin = self.plugin_manager.get(resource_type)


I'm not seeing any other references to ManagerService.plugin_manager in blazar. Is this supposed to be a call to self._get_plugin instead?

No idea how this got here, the branch is unreachable anyway since _get_plugin handles it already.

super-cooper · 2023-06-15T22:05:18Z

blazar/manager/service.py

-                    resource_type=resource_type)
-
-            plugin = self.plugins.get(resource_type)
+            plugin = self._get_plugin(resource_type)


self._get_plugin doesn't appear to have an appropriate check to throw UnsupportedResourceType and might result in confusing errors.

I am not sure what is better, unsupported resource type or invalid plugin name, which is raised right after if no plugin is returned. I left it as "invalid plugin name" to reduce the code base a bit.

super-cooper · 2023-06-15T22:06:36Z

blazar/manager/service.py

-
-            if resource_type not in self.plugins:
+            if resource_type in self.plugins:
+                plugin = self.plugins.get(resource_type)


Is there a reason why we're not using self._get_plugin here? It seems that most of this code is duplicated between the regular and third party plugin branches of this conditional. We could just use _get_plugin and most of the same code for both.

Good point. Like above I've similarly made this just raise "invalid plugin name"

super-cooper · 2023-06-16T20:06:47Z

blazar/plugins/third_party_plugins/base.py

+        policy.check_enforcement(self.resource_type(), "delete")
+        resource = db_api.resource_get(self.resource_type(), resource_id)
+        if resource is None:
+            raise manager_ex.ResourceNotFound(


This exception appears to come from plugin_ex, not manager_ex.

super-cooper · 2023-06-16T20:09:23Z

blazar/plugins/third_party_plugins/base.py

+
+    def api_update(self, resource_id, data):
+        policy.check_enforcement(self.resource_type(), "put")
+        extras = data["extras"]


Should this be data.get("extras")? I see we check for not extras in a couple lines

Yes, that's a good idea. I guess this would be the difference between not including it in the API request and it being empty. Either way, returning None is fine.

super-cooper · 2023-06-16T20:26:45Z

blazar/plugins/third_party_plugins/base.py

+        return rest
+
+    def query_resource_allocations(self, resources, lease_id=None,
+                                   reservation_id=None, detail=False):


detail appears unused

super-cooper · 2023-06-16T20:41:26Z

blazar/plugins/third_party_plugins/base.py

+        return resource_policy
+
+
+class ResourceMonitorPlugin():


Should this implement BaseMonitorPlugin?

super-cooper · 2023-06-16T21:21:33Z

blazar/plugins/third_party_plugins/stitchport_plugin.py

+                if port["data"]["port_id"] in active_port_ids
+            ]
+        except Exception as e:
+            LOG.exception('Skipping health check. %s', str(e))


This should return, no?

I would expect it to return empty lists in the next statement, this is copied from the host monitor.

Mark-Powers added 10 commits November 15, 2021 20:50

Add third party plugin support

276ab48

Merge branch 'chameleoncloud/train' of https://github.com/chameleoncl…

34f7da3

…oud/blazar into third_party_plugins

Continue adding third party plugin support

7db8a66

Add resources endpoint

54ceb91

Fix host plugin typos

dfa7da4

Fix third party host plugin

cc6016f

Fix formatting

aeb9e59

Update third party host plugin

853574d

Refactor host plugin

e754baa

Clean up DummyPlugin

35e2a2e

Mark-Powers closed this Dec 15, 2021

Mark-Powers reopened this Dec 15, 2021

Mark-Powers added 16 commits January 12, 2022 21:50

Revert "[backport] Prevent failed lease update from setting lease sta…

97f91eb

…tus to ERROR" This reverts commit d78c38a. Change-Id: Ia80e00a1aac3ad0c02a5062e997447fe243bacc6

Restore lease status after non fatal update exception

b98e8ef

Change-Id: Iaa8a88fb96d32c0bfe351af6115550bef5f80889

Merge branch 'chameleoncloud/train' into third_party_plugins

29d3f4b

Change-Id: I8263254dc7710d57bd50cd37ea3da2485dbbb38a

Update third party plugins with docs, tests

3d1c210

Change-Id: Ib1a06802ba827b2159de4fc61c97f2df273aff36

Add third party plugin support

2e4b883

Continue adding third party plugin support

1c208f1

Add resources endpoint

cc43b3e

Fix host plugin typos

5d740ab

Fix third party host plugin

b671d4a

Fix formatting

b11f78d

Update third party host plugin

ef50627

Refactor host plugin

844b0c6

Clean up DummyPlugin

cc52958

Update third party plugins with docs, tests

19b897b

Change-Id: Ib1a06802ba827b2159de4fc61c97f2df273aff36

Fix migrations and resource queries

f21a5ee

Change-Id: Idb31b4a048d6499cab6bbe1b90b93599c519a442

Fix broken checkouts

972b2db

Change-Id: Ic9bc1c9645aee9c7a98fd1d8ecb8f0169404161d

Mark-Powers marked this pull request as ready for review January 25, 2022 20:09

Mark-Powers requested a review from diurnalist January 25, 2022 20:09

Mark-Powers added 3 commits January 27, 2022 22:50

Add project_id as stitchport tag

1138230

Change-Id: I9f0d7f47e92ed222cbb7cd042304317f6c2e74ee

Fix pep8

d3480ab

Change-Id: Ia0bd2e39327b42597eae60a54a3050cc6ff0764c

Update stitchport plugin to use binding:profile

bb34404

Change-Id: I714c7389067e7d6d625790a3f776a07484652cb9

Mark-Powers removed the request for review from diurnalist August 4, 2022 15:47

Mark-Powers changed the base branch from chameleoncloud/train to chameleoncloud/xena June 1, 2023 15:30

Merge branch 'chameleoncloud/xena' into third_party_plugins

fd65787

Mark-Powers added 6 commits June 1, 2023 19:25

Merge branch 'chameleoncloud/xena' into third_party_plugins

9bdf28b

Fix git issues

8e6acf7

Merge branch 'third_party_plugins' of https://github.com/chameleonclo…

9e5b632

…ud/blazar into third_party_plugins

Fix git conflicts

ea556b5

Get third party plugins to work

7503f18

Finish TODOs

83d6bdd

Mark-Powers requested a review from super-cooper June 12, 2023 19:53

super-cooper suggested changes Jun 16, 2023

View reviewed changes

Update code based on review

216b203

Add third party plugins #54

Are you sure you want to change the base?

Add third party plugins #54

Uh oh!

Conversation

Mark-Powers commented Nov 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mark-Powers commented Jun 1, 2023

Uh oh!

super-cooper left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Mark-Powers commented Nov 30, 2021 •

edited

Loading