Unable to create a pdb: ORDSERROR

[rbaumgar]

After creating a cdb, by fixing the problems #126, #124, ...
I have a working ORDS server.

When I try to create a pdb, like the examples the pdb is in status: false

$ oc get pdb pdbx1 -o json|jq .status

{
  "action": "CREATE",
  "msg": "Getting PDB state",
  "openMode": "UNKNOWN",
  "phase": "Creating",
  "status": false,
  "totalSize": "1G"
}

The operator log

2024-08-22T07:08:19Z	INFO	pdb-webhook	Setting default values in PDB spec for : pdb1
2024-08-22T07:08:19Z	INFO	pdb-webhook	 - reuseTempFile : true
2024-08-22T07:08:19Z	INFO	pdb-webhook	 - unlimitedStorage : true
2024-08-22T07:08:19Z	INFO	pdb-webhook	 - tdeImport : false
2024-08-22T07:08:19Z	INFO	pdb-webhook	 - tdeExport : false
2024-08-22T07:08:19Z	INFO	pdb-webhook	 - asClone : false
2024-08-22T07:08:19Z	INFO	pdb-webhook	 - getScript : false
2024-08-22T07:08:19Z	INFO	pdb-webhook	ValidateCreate-Validating PDB spec for : pdb1
2024-08-22T07:08:19Z	INFO	pdb-webhook	validateCommon	{"name": "pdb1"}
2024-08-22T07:08:19Z	INFO	pdb-webhook	Valdiating PDB Resource Action : CREATE
2024-08-22T07:08:19Z	INFO	pdb-webhook	PDB Resource : pdb1 successfully validated for Action : CREATE
2024-08-22T07:08:19Z	INFO	controllers.PDB	Reconcile requested	{"multitenantoperator": {"name":"pdb1","namespace":"oracle"}}
2024-08-22T07:08:19Z	INFO	controllers.PDB	Adding finalizer	{"managePDBDeletion": {"name":"pdb1","namespace":"oracle"}}
2024-08-22T07:08:19Z	INFO	pdb-webhook	Setting default values in PDB spec for : pdb1
2024-08-22T07:08:19Z	INFO	pdb-webhook	ValidateUpdate-Validating PDB spec for : pdb1
2024-08-22T07:08:19Z	INFO	pdb-webhook	validateCommon	{"name": "pdb1"}
2024-08-22T07:08:19Z	INFO	pdb-webhook	Valdiating PDB Resource Action : CREATE
2024-08-22T07:08:19Z	INFO	controllers.PDB	Found PDB: pdb1	{"checkDuplicatePDB": "oracle"}
2024-08-22T07:08:19Z	INFO	controllers.PDB	Validating PDB phase for: pdb1	{"validatePhase": {"name":"pdb1","namespace":"oracle"}, "Action": "CREATE"}
2024-08-22T07:08:19Z	INFO	controllers.PDB	Validation complete	{"validatePhase": {"name":"pdb1","namespace":"oracle"}}
2024-08-22T07:08:19Z	INFO	controllers.PDB	PDB:	{"multitenantoperator": {"name":"pdb1","namespace":"oracle"}, "Name": "pdb1", "Phase": "Creating", "Status": "false"}
2024-08-22T07:08:19Z	INFO	controllers.PDB	Found CR for CDB	{"getCDBResource": {"name":"pdb1","namespace":"oracle"}, "Name": "cdb-dev", "CR Name": "cdb-dev"}
2024-08-22T07:08:19Z	INFO	controllers.PDB	Found CR for CDB	{"getCDBResource": {"name":"pdb1","namespace":"oracle"}, "Name": "cdb-dev", "CR Name": "cdb-dev"}
2024-08-22T07:08:19Z	INFO	controllers.PDB	Issuing REST call	{"callAPI": {"name":"pdb1","namespace":"oracle"}, "URL": "https://cdb-dev-ords.oracle:8888/ords/_/db-api/latest/database/pdbs/pdbnew/status", "Action": "GET"}
2024-08-22T07:08:23Z	DEBUG	events	cdb-dev	{"type": "Warning", "object": {"kind":"PDB","namespace":"oracle","name":"pdb1","uid":"7a7a01dc-26af-410b-9d9d-b29cb5e3f387","apiVersion":"database.oracle.com/v1alpha1","resourceVersion":"2535019946"}, "reason": "Done"}
2024-08-22T07:08:23Z	INFO	controllers.PDB	Issuing REST call	{"callAPI": {"name":"pdb1","namespace":"oracle"}, "URL": "https://cdb-dev-ords.oracle:8888/ords/_/db-api/latest/database/pdbs/", "Action": "POST"}
2024-08-22T07:08:28Z	ERROR	controllers.PDB	callAPI error	{"createPDB": {"name":"pdb1","namespace":"oracle"}, "err": "ORDS Error", "error": "ORDS Error"}
github.com/oracle/oracle-database-operator/controllers/database.(*PDBReconciler).createPDB
	/workspace/controllers/database/pdb_controller.go:695
github.com/oracle/oracle-database-operator/controllers/database.(*PDBReconciler).Reconcile
	/workspace/controllers/database/pdb_controller.go:225
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/root/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/root/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/root/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/root/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2024-08-22T07:08:28Z	INFO	controllers.PDB	Reconcile completed	{"multitenantoperator": {"name":"pdb1","namespace":"oracle"}}
2024-08-22T07:08:28Z	DEBUG	events	cdb-dev	{"type": "Warning", "object": {"kind":"PDB","namespace":"oracle","name":"pdb1","uid":"7a7a01dc-26af-410b-9d9d-b29cb5e3f387","apiVersion":"database.oracle.com/v1alpha1","resourceVersion":"2535020058"}, "reason": "Done"}

The ORDS log is empty.

The pdb CRD

#
# Copyright (c) 2022, Oracle and/or its affiliates. All rights reserved.
# Licensed under the Universal Permissive License v 1.0 as shown at http://oss.oracle.com/licenses/upl.
#
apiVersion: database.oracle.com/v1alpha1
kind: PDB
metadata:
  name: pdb1
  namespace: oracle
spec:
  cdbResName: "cdb-dev"
  cdbName: "devcdb"
  pdbName: "pdbnew"
  adminName:
    secret:
      secretName: "pdb1-secret"
      key: "sysadmin_user"
  adminPwd:
    secret:
      secretName: "pdb1-secret"
      key: "sysadmin_pwd"
  fileNameConversions: "NONE"
  totalSize: "1G"
  tempSize: "100M"
  action: "Create"
  ### added
  cdbNamespace: oracle  
  pdbTlsKey:
    secret:
      key: tls.key
      secretName: db-tls
  pdbTlsCrt:
    secret:
      key: tls.crt
      secretName: db-tls
  pdbTlsCat:
    secret:
      key: ca.crt
      secretName: db-ca 
  webServerUser:
    secret:
      key: webserver_user
      secretName: cdb1-secret
  webServerPwd:
    secret:
      key: webserver_pwd
      secretName: cdb1-secret        

[rbaumgar]

I called the ORDS REST API directly successfully.
I was able to create a pdb with the following payload

{
  "new_pdb_name": "pdb_sample",
  "admin_user": "<username>",
  "admin_password": "<password>",
  "file_name_convert": "NONE",
  "temp_file_reuse": true,
  "storage": "(MAXSIZE 2G MAX_SHARED_TEMP_SIZE 800M)",
  "dryrun": true
}

How to configure the pdb CRD to get the same REST API call by the operator?

[rbaumgar]

it looks like the parameter new_pdn_name is not provided by the operator, so it doesn't work with ORDS 24.2.

oracle-database-operator/controllers/database/pdb_controller.go

Lines 655 to 686 in 577a467

	values := map[string]string{
	"method": "CREATE",
	"pdb_name": pdb.Spec.PDBName,
	"adminName": pdbAdminName,
	"adminPwd": pdbAdminPwd,
	"fileNameConversions": pdb.Spec.FileNameConversions,
	"reuseTempFile": strconv.FormatBool(*(pdb.Spec.ReuseTempFile)),
	"unlimitedStorage": strconv.FormatBool(*(pdb.Spec.UnlimitedStorage)),
	"totalSize": pdb.Spec.TotalSize,
	"tempSize": pdb.Spec.TempSize,
	"getScript": strconv.FormatBool(*(pdb.Spec.GetScript))}
	if *(pdb.Spec.TDEImport) {
	tdePassword, err = r.getSecret(ctx, req, pdb, pdb.Spec.TDEPassword.Secret.SecretName, pdb.Spec.TDEPassword.Secret.Key)
	if err != nil {
	return err
	}
	tdeSecret, err = r.getSecret(ctx, req, pdb, pdb.Spec.TDESecret.Secret.SecretName, pdb.Spec.TDESecret.Secret.Key)
	if err != nil {
	return err
	}
	tdeSecret = tdeSecret[:len(tdeSecret)-1]
	tdePassword = tdeSecret[:len(tdePassword)-1]
	values["tdePassword"] = tdePassword
	values["tdeKeystorePath"] = pdb.Spec.TDEKeystorePath
	values["tdeSecret"] = tdeSecret
	}
	//url := "https://"+ pdb.Spec.CDBNamespace + "." + pdb.Spec.CDBResName + "-ords:" + strconv.Itoa(cdb.Spec.ORDSPort) + "/ords/_/db-api/latest/database/pdbs/"
	url := "https://" + pdb.Spec.CDBResName + "-ords." + pdb.Spec.CDBNamespace + ":" + strconv.Itoa(cdb.Spec.ORDSPort) + "/ords/_/db-api/latest/database/pdbs/"

[mmalvezz]

Yes correct it's not working with ords 24.... see the latest version of Dockerfile which uses earlier version

[rbaumgar]

@mmalvezz which dockerfile are you talking about?
Is a new operator version available?
Are new images available?

[mmalvezz]

Please have a look at https://github.com/oracle/oracle-database-operator/blob/main/ords/Dockerfile, just recreate the cdb image usng this one
....
....

FROM  container-registry.oracle.com/java/jdk:latest

# Environment variables required for this build (do NOT change)
# -------------------------------------------------------------
ENV ORDS_HOME=/opt/oracle/ords/ \
    RUN_FILE="runOrdsSSL.sh" \
    ORDSVERSION=23.4.0-8

# Copy binaries

[ilfur]

Since a week or so, my ORDS 23.4 cannot connect to my 23ai CDB anymore, TNS Errors (no service specified in connect and no default service configured)
This problem popped up with the 23.4 ORDS configured by OraOperator for PDB management
AND with a standalone ORDS 23.4 running Database Actions, APEX and more.
So I needed to upgrade my ORDS to 24.x to work again.
Now I cannot provision PDBs anymore due to the already known changed create_pdb syntax in the the internal REST call.
PLEASE fix this ASAP, should be no problem to use old AND new syntax for compatibility reasons.

[mmalvezz]

Let me check on my env with the ords 23.4 and 23ai... ords 24 is not in the roadmap.

[mmalvezz]

I'm testing ords 23.4 and rdbms 23ai...it works

kubectl get pods -n cdbnamespace
NAME                    READY   STATUS    RESTARTS   AGE
cdb-dev-ords-rs-xqzl9   1/1     Running   0          19m

kubectl logs -f cdb-dev-ords-rs-xqzl9 -n cdbnamespace
[...]
Completed verifying Oracle REST Data Services schema version 23.4.0.r3461619.
Connecting to database user: ORDS_PUBLIC_USER url: jdbc:oracle:thin:@(DESCRIPTION=(CONNECT_TIMEOUT=5)(TRANSPORT_CONNECT_TIMEOUT=3)(RETRY_COUNT=3)(ADDRESS_LIST=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=10.0.40.102)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=DB0207_c6t_lin.vncpod.vcndns.oraclevcn.com)))
The setting named: db.serviceNameSuffix was set to: .vncpod.vcndns.oraclevcn.com in configuration: default
The setting named: db.username was set to: ORDS_PUBLIC_USER in configuration: default
The setting named: db.password was set to: ****** in configuration: default
The setting named: security.requestValidationFunction was set to: ords_util.authorize_plsql_gateway in configuration: default

on database

select banner_legacy from v$version;

BANNER_LEGACY
--------------------------------------------------------------------------------
Oracle Database 23ai EE High Perf Release 23.0.0.0.0 - for Oracle Cloud and Engi neered Systems

SQL> select unique(machine) from v$session;

MACHINE
----------------------------------------------------------------
opr
cdb-dev-ords-rs-xqzl9
mmalvezzpcd

kubectl apply -f create_pdb1_resource.yaml
[...]
 kubectl get pdb pdb1 -n pdbnamespace
NAME   CDB NAME   PDB NAME   PDB STATE    PDB SIZE   STATUS   MESSAGE   CONNECT_STRING
pdb1   DB12       pdbdev     READ WRITE   0.96G      Ready    Success   (DESCRIPTION=(CONNECT_TIMEOUT=5)(TRANSPORT_CONNECT_TIMEOUT=3)(RETRY_COUNT=3)(ADDRESS_LIST=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=10.0.40.102)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=pdbdev.vncpod.vcndns.oraclevcn.com)))

on database

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDBTEST                        READ WRITE NO
         4 PDBDEV                         READ WRITE NO

[ilfur]

I think I may have found the problem. I was wondering where You got the TNS connect string from in Your logs.
In my env, I am using the old dbPort/dbServer/serviceName Syntax (OraOperator 1.0 CRD) to connect to the CDB.
You are probably using the dbTnsurl Parameter (I think new in 1.1) to specify a whole connect string with service name and everything else inside.

I cannot go back to version 23.4 anymore to test this, so I will wait for the newer OraOperator 1.2 instead to get free from ORDS version dependencies...