Skip to content

OraOperator v1.2.0 oracle-database-operator-controller-manager pods crashes #183

New issue
New issue
Open
Open
OraOperator v1.2.0 oracle-database-operator-controller-manager pods crashes#183
Assignees
rahilb1
@andbos

Description

@andbos
andbos
opened on Jul 3, 2025

OraOperator v1.2.0 oracle-database-operator-controller-manager crashes. Didn't have any similar problems before (v1.1.0).

AWS EKS v1.29.3.

Error:
2025-07-03T09:38:15Z ERROR controllers.database.SingleInstanceDatabase admission webhook "vsingleinstancedatabase.kb.io" denied the request: SingleInstanceDatabase.database.oracle.com "sidb11" is invalid: [spec.initParams.sgaTarget: Forbidden: cannot be changed, spec.initParams.pgaAggregateTarget: Forbidden: cannot be changed, spec.initParams.cpuCount: Forbidden: cannot be changed, spec.initParams.processes: Forbidden: cannot be changed] {"manageSingleInstanceDatabaseDeletion": {"name":"sidb11","namespace":"oracle-database"}, "error": "admission webhook "vsingleinstancedatabase.kb.io" denied the request: SingleInstanceDatabase.database.oracle.com "sidb11" is invalid: [spec.initParams.sgaTarget: Forbidden: cannot be changed, spec.initParams.pgaAggregateTarget: Forbidden: cannot be changed, spec.initParams.cpuCount: Forbidden: cannot be changed, spec.initParams.processes: Forbidden: cannot be changed]"}

2025-07-03T09:38:15Z    ERROR   controllers.database.SingleInstanceDatabase     admission webhook "vsingleinstancedatabase.kb.io" denied the request: SingleInstanceDatabase.database.oracle.com "sidb11" is invalid: [spec.initParams.sgaTarget: Forbidden: cannot be changed, spec.initParams.pgaAggregateTarget: Forbidden: cannot be changed, spec.initParams.cpuCount: Forbidden: cannot be changed, spec.initParams.processes: Forbidden: cannot be changed]  {"manageSingleInstanceDatabaseDeletion": {"name":"sidb11","namespace":"oracle-database"}, "error": "admission webhook \"vsingleinstancedatabase.kb.io\" denied the request: SingleInstanceDatabase.database.oracle.com \"sidb11\" is invalid: [spec.initParams.sgaTarget: Forbidden: cannot be changed, spec.initParams.pgaAggregateTarget: Forbidden: cannot be changed, spec.initParams.cpuCount: Forbidden: cannot be changed, spec.initParams.processes: Forbidden: cannot be changed]"}
github.com/oracle/oracle-database-operator/controllers/database.(*SingleInstanceDatabaseReconciler).manageSingleInstanceDatabaseDeletion
        /workspace/controllers/database/singleinstancedatabase_controller.go:3151
github.com/oracle/oracle-database-operator/controllers/database.(*SingleInstanceDatabaseReconciler).Reconcile
        /workspace/controllers/database/singleinstancedatabase_controller.go:158
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
        /root/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
        /root/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:303
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
        /root/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:263
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
        /root/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:224
2025-07-03T09:38:15Z    INFO    controllers.database.SingleInstanceDatabase     Reconcile queued
E0703 09:38:15.257430       1 panic.go:262] "Observed a panic" panic="runtime error: invalid memory address or nil pointer dereference" panicGoValue="\"invalid memory address or nil pointer dereference\"" stacktrace=<
        goroutine 972 [running]:
        k8s.io/apimachinery/pkg/util/runtime.logPanic({0x2f80748, 0x45dcfa0}, {0x25978e0, 0x457aac0})
                /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:107 +0xbc
        k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x2f80748, 0x45dcfa0}, {0x25978e0, 0x457aac0}, {0x45dcfa0, 0x0, 0x43aaa5?})
                /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:82 +0x5e
        k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000cc0e00?})
                /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:59 +0x108
        panic({0x25978e0?, 0x457aac0?})
                /usr/local/go/src/runtime/panic.go:785 +0x132
        github.com/oracle/oracle-database-operator/controllers/database.(*ShardingDatabaseReconciler).SetupWithManager.(*ShardingDatabaseReconciler).eventFilterPredicate.func2({{0x2faaf78?, 0xc00084c8c0?}, {0x2faaf78?, 0xc0002dbb80?}})
                /workspace/controllers/database/shardingdatabase_controller.go:462 +0xb5
        sigs.k8s.io/controller-runtime/pkg/predicate.TypedFuncs[...].Update(...)
                /root/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/predicate/predicate.go:95
        sigs.k8s.io/controller-runtime/pkg/internal/source.(*EventHandler[...]).OnUpdate(0x2f92600, {0x29d1760?, 0xc00084c8c0}, {0x29d1760, 0xc0002dbb80})
                /root/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/event_handler.go:117 +0x3ba
        k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
                /root/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:253
        k8s.io/client-go/tools/cache.(*processorListener).run.func1()
                /root/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:976 +0xea
        k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
                /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
        k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0004f2f70, {0x2f5ba00, 0xc0012940c0}, 0x1, 0xc000812fc0)
                /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
        k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000afef70, 0x3b9aca00, 0x0, 0x1, 0xc000812fc0)
                /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
        k8s.io/apimachinery/pkg/util/wait.Until(...)
                /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
        k8s.io/client-go/tools/cache.(*processorListener).run(0xc0000e2090)
                /root/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:972 +0x5a
        k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
                /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x4c
        created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 375
                /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x73
 >
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x21de475]

goroutine 972 [running]:
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x2f80748, 0x45dcfa0}, {0x25978e0, 0x457aac0}, {0x45dcfa0, 0x0, 0x43aaa5?})
        /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:89 +0xee
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000cc0e00?})
        /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:59 +0x108
panic({0x25978e0?, 0x457aac0?})
        /usr/local/go/src/runtime/panic.go:785 +0x132
github.com/oracle/oracle-database-operator/controllers/database.(*ShardingDatabaseReconciler).SetupWithManager.(*ShardingDatabaseReconciler).eventFilterPredicate.func2({{0x2faaf78?, 0xc00084c8c0?}, {0x2faaf78?, 0xc0002dbb80?}})
        /workspace/controllers/database/shardingdatabase_controller.go:462 +0xb5
sigs.k8s.io/controller-runtime/pkg/predicate.TypedFuncs[...].Update(...)
        /root/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/predicate/predicate.go:95
sigs.k8s.io/controller-runtime/pkg/internal/source.(*EventHandler[...]).OnUpdate(0x2f92600, {0x29d1760?, 0xc00084c8c0}, {0x29d1760, 0xc0002dbb80})
        /root/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/source/event_handler.go:117 +0x3ba
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)
        /root/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:253
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /root/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:976 +0xea
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000aabf70, {0x2f5ba00, 0xc0012940c0}, 0x1, 0xc000812fc0)
        /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000afef70, 0x3b9aca00, 0x0, 0x1, 0xc000812fc0)
        /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0000e2090)
        /root/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:972 +0x5a
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:72 +0x4c
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 375
        /root/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:70 +0x73

$ kubectl -n oracle-database-operator-system get pods
NAME                                                           READY   STATUS             RESTARTS      AGE
oracle-database-operator-controller-manager-59c5c555d6-jncvj   0/1     Error              4 (78s ago)   4m35s
oracle-database-operator-controller-manager-59c5c555d6-m8vb4   0/1     CrashLoopBackOff   4 (44s ago)   4m35s
oracle-database-operator-controller-manager-59c5c555d6-q4hnx   0/1     CrashLoopBackOff   4 (28s ago)   4m35s

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions