This integration collects telemetry from Databricks (including Spark on Databricks) and/or Spark telemetry from any Spark deployment. See the Features section for supported telemetry types.

• Important Notes
• Getting Started
  • On-host Deployment
• Features
• Usage
  • Command Line Options
  • Configuration
    • General configuration
    • Pipeline configuration
    • Log configuration
    • Databricks configuration
    • Spark configuration
  • Authentication
  • Apache Spark Data
  • Consumption & Cost Data
  • Job Run Data
  • Pipeline Update Metrics
  • Pipeline Event Logs
  • Cluster Health
  • Query Metrics
• Building
  • Coding Conventions
  • Local Development
  • Releases
  • Github Workflows
• Appendix

• All references within this document to Databricks documentation reference the Databricks on AWS documentation. Use the cloud switcher menu located in the upper right hand corner of the documentation to select corresponding documentation for a different cloud provider.

To get started with the Databricks Integration, deploy the integration using a supported deployment type, configure the integration using supported configuration mechanisms, and optionally import the sample dashboards included in the examples directory.

The Databricks Integration can be deployed inside a Databricks cluster on the cluster's driver node (recommended) or outside a Databricks cluster on a supported host platform.

NOTE:

• Apache Spark data for a Databricks cluster cannot be collected remotely. In order to collect Apache Spark data from a Databricks cluster, the Databricks Integration must be deployed inside the cluster.
• The Databricks Integration can be used separately from Databricks to solely collect Apache Spark data from any accessible Spark deployment.

The Databricks Integration can be deployed on the driver node of a Databricks all-purpose, job, or pipeline cluster using a cluster-scoped init script. The init script uses custom environment variables to specify configuration parameters necessary for the integration configuration.

To install the init script on an all-purpose cluster, perform the following steps.

1. Login to your Databricks account and navigate to the desired workspace.
2. Follow the recommendations for init scripts to store the cluster_init_integration.sh script within your workspace in the recommended manner. For example, if your workspace is enabled for Unity Catalog, you should store the init script in a Unity Catalog volume.
3. Navigate to the Compute tab and select the desired all-purpose compute to open the compute details UI.
4. Click the button labeled Edit to edit the compute's configuration.
5. Follow the steps to use the UI to configure a cluster-scoped init script and point to the location where you stored the init script in step 2 above.
6. If your cluster is not running, click on the button labeled Confirm to save your changes. Then, restart the cluster. If your cluster is already running, click on the button labeled Confirm and restart to save your changes and restart the cluster.

Additionally, follow the steps to set environment variables to add the following environment variables.

• NEW_RELIC_API_KEY - Your New Relic User API Key
• NEW_RELIC_LICENSE_KEY - Your New Relic License Key
• NEW_RELIC_ACCOUNT_ID - Your New Relic Account ID
• NEW_RELIC_REGION - The region of your New Relic account; one of US or EU
• NEW_RELIC_DATABRICKS_WORKSPACE_HOST - The instance name of the target Databricks instance
• NEW_RELIC_DATABRICKS_ACCESS_TOKEN - To authenticate with a personal access token, your personal access token
• NEW_RELIC_DATABRICKS_OAUTH_CLIENT_ID - To use a service principal to authenticate with Databricks (OAuth M2M), the OAuth client ID for the service principal
• NEW_RELIC_DATABRICKS_OAUTH_CLIENT_SECRET - To use a service principal to authenticate with Databricks (OAuth M2M), an OAuth client secret associated with the service principal
• NEW_RELIC_DATABRICKS_USAGE_ENABLED - Set to true to enable collection of consumption and cost data from this cluster node or false to disable collection. Defaults to true.
• NEW_RELIC_DATABRICKS_SQL_WAREHOUSE - The ID of a SQL warehouse where usage queries should be run
• NEW_RELIC_DATABRICKS_JOB_RUNS_ENABLED - Set to true to enable collection of job run data from this cluster node or false to disable collection. Defaults to true.
• NEW_RELIC_DATABRICKS_PIPELINE_EVENT_LOGS_ENABLED - Set to true to enable collection of Databricks Delta Live Tables pipeline event logs from this cluster node or false to disable collection. Defaults to true.
• NEW_RELIC_DATABRICKS_QUERY_METRICS_ENABLED - Set to true to enable collection of Databricks query metrics for this workspace or false to disable collection. Defaults to true.
• NEW_RELIC_INFRASTRUCTURE_ENABLED - Set to true to install the New Relic Infrastructure agent on the driver and worker nodes of the cluster. Defaults to false.
• NEW_RELIC_INFRASTRUCTURE_LOGS_ENABLED - Set to true to enable collection of the Spark driver and executor logs, the Spark driver event log, and the driver and worker init script logs. Logs will be forwarded to New Relic Logs via the New Relic Infrastructure agent and therefore, setting this environment variable to true requires that the NEW_RELIC_INFRASTRUCTURE_ENABLED environment variable also be set to true. Defaults to false.

NOTE:

• The NEW_RELIC_API_KEY and NEW_RELIC_ACCOUNT_ID are currently unused but are required by the new-relic-client-go module used by the integration.
• Only the personal access token or OAuth credentials need to be specified but not both. If both are specified, the OAuth credentials take precedence.
• Sensitive data like credentials and API keys should never be specified directly in custom environment variables. Instead, it is recommended to create a secret using the Databricks CLI and reference the secret in the environment variable. See the appendix for an example of creating a secret and referencing it in a custom environment variable.
• To install the init script on a job cluster, see the section configure compute for jobs in the Databricks documentation.
• To install the init script on a pipeline cluster, see the section configure compute for a DLT pipeline in the Databricks documentation.
• Make sure to restart the cluster following the configuration of the environment variables.

The Databricks Integration provides binaries for the following host platforms.

• Linux amd64
• Windows amd64

To run the Databricks integration on a host outside a Databricks cluster, perform the following steps.

1. Download the appropriate archive for your platform from the latest release.
2. Extract the archive to a new or existing directory.
3. Create a directory named configs in the same directory.
4. Create a file named config.yml in the configs directory and copy the contents of the file configs/config.template.yml in this repository into it.
5. Edit the config.yml file to configure the integration appropriately for your environment.
6. From the directory where the archive was extracted, execute the integration binary using the command ./newrelic-databricks-integration (or .\newrelic-databricks-integration.exe on Windows) with the appropriate Command Line Options.

The Databricks Integration supports the following capabilities.

• Collect Spark telemetry

  When deployed inside a Databricks cluster, the Databricks Integration can collect telemetry from all Spark applications running in the cluster.

  When deployed outside a Databricks cluster, the Databricks Integration can also collect Spark telemetry from all Spark applications on any accessible Spark deployment.

• Collect Databricks consumption and cost data

  The Databricks Integration can collect consumption and cost related data from the Databricks system tables. This data can be used to show Databricks DBU consumption metrics and estimated Databricks costs directly within New Relic.

• Collect Databricks job run telemetry

  The Databricks Integration can collect telemetry about Databricks Job runs, such as job run durations, task run durations, the current state of job and task runs, if a job or a task is a retry, and the number of times a task was retried.

• Collect Databricks Delta Live Tables Pipeline event logs

  The Databricks Integration can collect Databricks Delta Live Tables Pipeline event logs for all Databricks Delta Live Tables Pipelines defined in a workspace. Databricks Delta Live Tables Pipeline event log entries for every pipeline update are collected and sent to New Relic Logs.

The Databricks integration is configured using the config.yml and/or environment variables. For Databricks, authentication related configuration parameters may also be set in a Databricks configuration profile. In all cases, where applicable, environment variables always take precedence.

All configuration parameters for the Databricks integration can be set using a YAML file named config.yml. The default location for this file is configs/config.yml relative to the current working directory when the integration binary is executed. The supported configuration parameters are listed below. See config.template.yml for a full configuration example.

The parameters in this section are configured at the top level of the config.yml.

This parameter specifies the New Relic License Key (INGEST) that should be used to send generated metrics.

The license key can also be specified using the NEW_RELIC_LICENSE_KEY environment variable.

This parameter specifies which New Relic region that generated metrics should be sent to.

This parameter specifies the interval (in seconds) at which the integration should poll for data.

This parameter is only used when runAsService is set to true.

The integration can run either as a "service" or as a simple command line utility which runs once and exits when it is complete.

When set to true, the integration process will run continuously and poll the for data at the recurring interval specified by the interval parameter. The process will only exit if it is explicitly stopped or a fatal error or panic occurs.

When set to false, the integration will run once and exit. This is intended for use with an external scheduling mechanism like cron.

The integration retrieves, processes, and exports data to New Relic using a data pipeline consisting of one or more receivers, a processing chain, and a New Relic exporter. Various aspects of the pipeline are configurable. This element groups together the configuration parameters related to pipeline configuration.

The integration uses the logrus package for application logging. This element groups together the configuration parameters related to log configuration.

The integration execution mode. Currently, the only supported execution mode is databricks.

Deprecated: As of v2.3.0, this configuration parameter is no longer used. The presence (or not) of the databricks top-level node will be used to enable (or disable) the Databricks collector. Likewise, the presence (or not) of the spark top-level node will be used to enable (or disable) the Spark collector separate from Databricks.

This element groups together the configuration parameters to configure the Databricks collector. If this element is not specified, the Databricks collector will not be run.

Note that this node is not required. It can be used with or without the spark top-level node.

This element groups together the configuration parameters to configure the Spark collector. If this element is not specified, the Spark collector will not be run.

Note that this node is not required. It can be used with or without the databricks top-level node.

This element specifies a group of custom tags that will be added to all telemetry sent to New Relic. The tags are specified as a set of key-value pairs.

This parameter specifies the size of the buffer that holds received items before being flushed through the processing chain and on to the exporters. When this size is reached, the items in the buffer will be flushed automatically.

This parameter specifies the interval (in seconds) at which the pipeline should automatically flush received items through the processing chain and on to the exporters. Each time this interval is reached, the pipeline will flush items even if the item buffer has not reached the size specified by the receiveBufferSize parameter.

The integration retrieves, processes, and exports metrics to New Relic using a data pipeline consisting of one or more receivers, a processing chain, and a New Relic exporter. When runAsService is true, the integration can launch one or more "instances" of this pipeline to receive, process, and export data concurrently. Each "instance" will be configured with the same processing chain and exporters and the receivers will be spread across the available instances in a round-robin fashion.

This parameter specifies the number of pipeline instances to launch.

NOTE: When runAsService is false, only a single pipeline instance is used.

This parameter specifies the maximum severity of log messages to output with trace being the least severe and panic being the most severe. For example, at the default log level (warn), all log messages with severities warn, error, fatal, and panic will be output but info, debug, and trace will not.

This parameter designates a file path where log output should be written. When no path is specified, log output will be written to the standard error stream (stderr).

The Databricks configuration parameters are used to configure the Databricks collector.

This parameter specifies the instance name of the target Databricks instance for which data should be collected. This is used by the integration when constructing the URLs for API calls. Note that the value of this parameter must not include the https:// prefix, e.g. https://my-databricks-instance-name.cloud.databricks.com.

The workspace host can also be specified using the DATABRICKS_HOST environment variable.

When set, the integration will use Databricks personal access token authentication to authenticate Databricks API calls with the value of this parameter as the Databricks personal access token.

The personal access token can also be specified using the DATABRICKS_TOKEN environment variable or any other SDK-supported mechanism (e.g. the token field in a Databricks configuration profile).

See the authentication section for more details.

When set, the integration will use a service principal to authenticate with Databricks (OAuth M2M) when making Databricks API calls. The value of this parameter will be used as the OAuth client ID.

The OAuth client ID can also be specified using the DATABRICKS_CLIENT_ID environment variable or any other SDK-supported mechanism (e.g. the client_id field in a Databricks configuration profile).

See the authentication section for more details.

When the oauthClientId is set, this parameter can be set to specify the OAuth secret associated with the service principal.

The OAuth client secret can also be specified using the DATABRICKS_CLIENT_SECRET environment variable or any other SDK-supported mechanism (e.g. the client_secret field in a Databricks configuration profile).

See the authentication section for more details.

Certain telemetry and data collected by the Databricks collector requires the collector to run Databricks SQL statements on a SQL warehouse. This configuration parameter specifies the number of seconds to wait before timing out a pending or running SQL query.

This element groups together the configuration parameters to configure the Databricks collector settings related to the collection of consumption and cost data.

This element groups together the configuration parameters to configure the Databricks collector settings related to the collection of job data.

This element groups together the configuration parameters to configure the Databricks collector settings related to the collection of Databricks Delta Live Tables Pipelines telemetry.

This element groups together the configuration parameters to configure the Databricks collector settings related to the collection of Databricks query telemetry.

The Databricks usage configuration parameters are used to configure Databricks collector settings related to the collection of Databricks consumption and cost data.

By default, when the Databricks collector is enabled, it will automatically collect consumption and cost data.

This flag can be used to disable the collection of consumption and cost data by the Databricks collector. This may be useful when running multiple instances of the Databricks Integration. In this scenario, Databricks consumption and cost data collection should only be enabled on a single instance. Otherwise, this data will be recorded more than once in New Relic, affecting consumption and cost calculations.

The ID of a SQL warehouse on which to run the SQL statements used to collect Databricks consumption and cost data.

This parameter is required when the collection of Databricks consumption and cost data is enabled.

When the collection of Databricks consumption and cost data is enabled, the Databricks collector can include several pieces of identifying information along with the consumption and cost data.

By default, when the collection of Databricks consumption and cost data is enabled, the Databricks collector will not collect such data as it may be personally identifiable. This flag can be used to enable the inclusion of the identifying information.

When enabled, the following values are included.

• The identity of the user a serverless billing record is attributed to. This value is included in the identity metadata returned from usage records in the billable usage system table.
• The identity of the cluster creator for each usage record for billable usage attributed to all-purpose and job compute.
• The single user name for each usage record for billable usage attributed to all-purpose and job compute configured for single-user access mode.
• The identity of the warehouse creator for each usage record for billable usage attributed to SQL warehouse compute.
• The identity of the user or service principal used to run jobs for each query result collected by job cost data queries.

This parameter specifies the time of day at which the collection of consumption and cost data occur. The value must be of the form HH:mm:ss where HH is the 0-padded 24-hour clock hour (00 - 23), mm is the 0-padded minute (00 - 59) and ss is the 0-padded second (00 - 59). For example, 09:00:00 is the time 9:00 AM and 23:30:00 is the time 11:30 PM.

The time will always be interpreted according to the UTC time zone. The time zone can not be configured. For example, to specify that the integration should be run at 2:00 AM EST (-0500), the value 07:00:00 should be specified.

When the collection of Databricks consumption and cost data is enabled, the Databricks collector will always collect billable usage data and list pricing data on every run. In addition, by default, the Databricks collector will also run all job cost queries on every run. However, the latter behavior can be configured using a set of flags specified with this configuration property to selectively enable or disable the job cost queries. Each flag is specified using a property with the query ID as the name of the property and true or false as the value of the property.The following flags are supported.

• jobs_cost_list_cost_per_job_run
• jobs_cost_list_cost_per_job
• jobs_cost_frequent_failures
• jobs_cost_most_retries

For example, to enable the list cost per job run query and the list cost per job query but disable the list cost of failed job runs for jobs with frequent failures query and the list cost of repaired job runs for jobs with frequent repairs query, the following configuration would be specified.

  optionalQueries:
    jobs_cost_list_cost_per_job_run: true
    jobs_cost_list_cost_per_job: true
    jobs_cost_frequent_failures: false
    jobs_cost_most_retries: false

This element groups together the configuration parameters to configure the Databricks collector settings related to the collection of job run data.

The Databricks job run configuration parameters are used to configure Databricks collector settings related to the collection of Databricks job run data.

By default, when the Databricks collector is enabled, it will automatically collect job run data.

This flag can be used to disable the collection of job run data by the Databricks collector. This may be useful when running multiple instances of the Databricks Integration against the same Databricks workspace. In this scenario, Databricks job run data collection should only be enabled on a single instance of the integration. Otherwise, this data will be recorded more than once in New Relic, affecting product features that use job run metrics (e.g. dashboards and alerts).

This parameter specifies a prefix that will be prepended to each Databricks job run metric name when the metric is exported to New Relic.

For example, if this parameter is set to databricks., then the full name of the metric representing the duration of a job run (job.run.duration) will be databricks.job.run.duration.

Note that it is not recommended to leave this value empty as the metric names without a prefix may be ambiguous.

By default, the Databricks collector will not include job run IDs on any of the job run metrics in order to avoid possible violations of metric cardinality limits due to the fact that job run IDs have high cardinality because they are unique across all jobs and job runs.

This flag can be used to enable the inclusion of the job run ID in the databricksJobRunId attribute on all job metrics.

When enabled, use the Limits UI and/or create a dashboard in order to monitor your limit status. Additionally, set alerts on resource metrics to provide updates on limits changes.

This parameter specifies an offset, in seconds that can be used to tune the collector's performance by limiting the number of job runs to return by constraining the start time of job runs to match to be greather than a particular time in the past calculated as an offset from the current time.

See the section startOffset Configuration for more details.

This element groups together the configuration parameters to configure the Databricks collector settings related to the collection of Databricks Delta Live Tables pipeline update metrics.

This element groups together the configuration parameters to configure the Databricks collector settings related to the collection of Databricks Delta Live Tables pipeline event logs.

The Databricks pipeline event logs configuration parameters are used to configure Databricks collector settings related to the collection of Databricks Delta Live Tables pipeline event logs.

By default, when the Databricks collector is enabled, it will automatically collect Databricks Delta Live Tables pipeline event logs.

This flag can be used to disable the collection of Databricks Delta Live Tables pipeline event logs by the Databricks collector. This may be useful when running multiple instances of the Databricks Integration against the same Databricks workspace. In this scenario, the collection of Databricks Delta Live Tables pipeline event logs should only be enabled on a single instance of the integration. Otherwise, duplicate New Relic log entries will be created for each Databricks Delta Live Tables pipeline event log entry, making troubleshooting challenging and affecting product features that rely on signal integrity such as anomaly detection.

The Databricks pipeline update metrics configuration parameters are used to configure Databricks collector settings related to the collection of Databricks Delta Live Tables pipeline update metrics.

By default, when the Databricks collector is enabled, it will automatically collect Databricks Delta Live Tables pipeline update metrics.

This flag can be used to disable the collection of Databricks Delta Live Tables pipeline update metrics by the Databricks collector. This may be useful when running multiple instances of the Databricks Integration against the same Databricks workspace. In this scenario, the collection of Databricks Delta Live Tables pipeline update metrics should only be enabled on a single instance of the integration. Otherwise, Databricks Delta Live Tables pipeline update metrics will be recorded more than once, making troubleshooting challenging and affecting product features that use these metrics (e.g. dashboards and alerts).

This parameter specifies a prefix that will be prepended to each Databricks pipeline update metric name when the metric is exported to New Relic.

For example, if this parameter is set to databricks., then the full name of the metric representing the duration of a pipeline update (pipeline.update.duration) will be databricks.pipeline.update.duration.

Note that it is not recommended to leave this value empty as the metric names without a prefix may be ambiguous.

By default, the Databricks collector will not include pipeline update IDs on any of the pipeline update metrics in order to avoid possible violations of metric cardinality limits due to the fact that update IDs have high cardinality because they are unique across all pipeline updates.

This flag can be used to enable the inclusion of the pipeline update ID in the databricksPipelineUpdateId attribute on all pipeline update metrics.

When enabled, use the Limits UI and/or create a dashboard in order to monitor your limit status. Additionally, set alerts on resource metrics to provide updates on limits changes.

This parameter specifies an offset, in seconds, that can be used to tune the collector's performance by limiting the number of pipeline events to return by constraining the timestamp of pipeline events to match to be greather than a particular time in the past calculated as an offset from the current time.

See the section startOffset Configuration for more details.

This parameter specifies an offset, in seconds, that the collector will use to delay collection of pipeline events in order to account for potential lag in the list pipeline events endpoint of the Databricks ReST API.

See the section intervalOffset Configuration for more details.

This element groups together the configuration parameters to configure the Databricks collector settings related to the collection of Databricks query metrics.

The Databricks query metrics configuration parameters are used to configure Databricks collector settings related to the collection of Databricks query metrics.

By default, when the Databricks collector is enabled, it will automatically collect Databricks query metrics.

This flag can be used to disable the collection of query metrics by the Databricks collector. This may be useful when running multiple instances of the Databricks Integration against the same Databricks workspace. In this scenario, the collection of Databricks query metrics should only be enabled on a single instance of the integration. Otherwise, query metrics will be recorded more than once, making troubleshooting challenging and affecting product features that use these metrics (e.g. dashboards and alerts).

When the collection of Databricks query metrics is enabled, the Databricks collector can include several pieces of identifying information along with the query metric data.

By default, when the collection of Databricks query metrics is enabled, the Databricks collector will not collect such data as it may be personally identifiable. This flag can be used to enable the inclusion of the identifying information.

When enabled, the following values are included.

• The ID of the user that executed the query
• The email address or username of the user that executed the query
• The ID of the user whose credentials were used to execute the query
• The email address or username whose credentials were used to execute the query
• The identity of the warehouse creator of the SQL warehouse where the query was executed

This parameter specifies an offset, in seconds, that the collector will use to calculate the start time of the time range used to call the list query history API.

See the section Query Metrics startOffset Configuration for more details.

This parameter specifies an offset, in seconds, that the collector will use to offset the current time window when checking for terminated queries, allowing the query end time additional time to be reflected in the list query history API.

See the section Query Metrics intervalOffset Configuration for more details.

This parameter specifies the maximum number of query results to return per page when listing queries via the list query history API

NOTE: For performance reasons, since the integration retrieves query metrics for each query using the list query history API, it is recommended to keep the maxResults parameter at its default value in most cases.

The Spark configuration parameters are used to configure the Spark collector.

This parameter can be used to monitor a Spark deployment. It specifies the URL of the Web UI of an application running on the Spark deployment to monitor. The value should be of the form http[s]://<hostname>:<port> where <hostname> is the hostname of the Spark deployment to monitor and <port> is the port number of the Spark application's Web UI (typically 4040 or 4041, 4042, etc if more than one application is running on the same host).

Note that the value must not contain a path. The path of the Spark ReST API endpoints (mounted at /api/v1) will automatically be appended.

This parameter specifies a prefix that will be prepended to each Spark metric name when the metric is exported to New Relic.

For example, if this parameter is set to spark., then the full name of the metric representing the value of the memory used on application executors (app.executor.memoryUsed) will be spark.app.executor.memoryUsed.

Note that it is not recommended to leave this value empty as the metric names without a prefix may be ambiguous.

This parameter specifies an identifier that indicates the cluster manager on which the Spark engine is running.

The valid values for this parameter are standalone and databricks. When this parameter is set to databricks, metrics are decorated with additional attributes (such as the workspace ID, name, and URL). Additional parameters that apply to Databricks can be specified using the Spark databricks configuration. When this parameter is not set, or is set to standalone, no additional telemetry or attributes are collected. The default value for this parameter is standalone.

This element groups together the configuration parameters to configure the Databricks specific settings when the Spark collector is collecting telemetry from Spark running on Databricks.

The Spark Databricks configuration parameters are used to configure the Databricks specific settings when the Spark collector is collecting telemetry from Spark running on Databricks.

By default, the Spark collector will not include task run IDs on any of the Spark job, stage, and task metrics associated with Databricks job runs in order to avoid possible violations of metric cardinality limits due to the fact that task run IDs have high cardinality because they are unique across all job task runs.

This flag can be used to enable the inclusion of the task run ID in the databricksJobRunTaskRunId attribute on all Spark job, stage, and task metrics associated with Databricks job runs.

When enabled, use the Limits UI and/or create a dashboard in order to monitor your limit status. Additionally, set alerts on resource metrics to provide updates on limits changes.

See the section "Mapping Spark metrics to Databricks Jobs and Pipelines" for more details.

By default, the Spark collector will not include pipeline update IDs on any of the Spark job, stage, and task metrics associated with Databricks pipeline updates in order to avoid possible violations of metric cardinality limits due to the fact that pipeline update IDs have high cardinality because they are unique across all pipeline updates.

This flag can be used to enable the inclusion of the pipeline update ID in the databricksPipelineUpdateId attribute on all Spark job, stage, and task metrics associated with Databricks pipeline updates.

When enabled, use the Limits UI and/or create a dashboard in order to monitor your limit status. Additionally, set alerts on resource metrics to provide updates on limits changes.

See the section "Mapping Spark metrics to Databricks Jobs and Pipelines" for more details.

By default, the Spark collector will not include pipeline update flow IDs on any of the Spark job, stage, and task metrics associated with Databricks pipeline update flows in order to avoid possible violations of metric cardinality limits due to the fact that pipeline update flow IDs have high cardinality because they unique across all pipeline updates.

This flag can be used to enable the inclusion of the pipeline update flow ID in the databricksPipelineFlowId attribute on all Spark job, stage, and task metrics associated with Databricks pipeline update flows.

When enabled, use the Limits UI and/or create a dashboard in order to monitor your limit status. Additionally, set alerts on resource metrics to provide updates on limits changes.

See the section "Mapping Spark metrics to Databricks Jobs and Pipelines" for more details.

The Databricks integration uses the Databricks SDK for Go to access the Databricks ReST API. The SDK performs authentication on behalf of the integration and provides many options for configuring the authentication type and credentials to be used. See the SDK documentation and the Databricks client unified authentication documentation for details.

For convenience purposes, the following parameters can be used in the Databricks configuration section of the `config.yml file.

• accessToken - When set, the integration will instruct the SDK to explicitly use Databricks personal access token authentication. The SDK will not attempt to try other authentication mechanisms and instead will fail immediately if personal access token authentication fails.
• oauthClientId - When set, the integration will instruct the SDK to explicitly use a service principal to authenticate with Databricks (OAuth M2M). The SDK will not attempt to try other authentication mechanisms and instead will fail immediately if OAuth M2M authentication fails. The OAuth Client secret can be set using the oauthClientSecret configuration parameter or any of the other mechanisms supported by the SDK (e.g. the client_secret field in a Databricks configuration profile or the DATABRICKS_CLIENT_SECRET environment variable).
• oauthClientSecret - The OAuth client secret to use for OAuth M2M authentication. This value is only used when oauthClientId is set in the config.yml. The OAuth client secret can also be set using any of the other mechanisms supported by the SDK (e.g. the client_secret field in a Databricks configuration profile or the DATABRICKS_CLIENT_SECRET environment variable).

NOTE: Authentication applies only to calls made to collect Databricks telemetry via the Databricks SDK for Go. The Spark collector does not support authentication when accessing the Spark ReST API.

The Databricks Integration can collect Apache Spark application metrics for all running applications in a given Spark cluster. Metrics are collected by accessing the monitoring ReST API through the Web UI of a given SparkContext.

This feature can be used for any Spark cluster, not just for Spark running on Databricks.

When the Spark collector is enabled (the top-level spark node is specified), Spark application metrics are collected from the Web UI URL specified in the webUiUrl or from the Web UI URL https://localhost:4040 by default.

Spark application telemetry is sent to New Relic as dimensional metrics. The provided metrics and attributes (dimensions) are listed in the sections below.

NOTE: Many of the descriptions below are sourced from the Apache Spark monitoring ReST API documentation.

The Apache Spark monitoring ReST API returns two types of metrics: monotonically increasing counters (referred to below simply as "counters") and gauges, with the majority of metrics being counters.

While all gauge metrics returned from the ReST API are created as gauge metrics within New Relic, note that all counter metrics returned from the ReST API are also created as gauge metrics. While the latest(), min(), max(), sum(), count(), and average() NRQL aggregator functions are all therefore available for use with these metrics, only latest() will provide meaningful results (e.g. taking the average of a set of data points that represent a monotonically increasing counter makes no sense).

Therefore, in general, only the latest() aggregator function should be used when visualizing or alerting on metrics listed in the sections below with the metric type counter.

Further to this, even when using the latest() aggregator function, the metrics have no meaning without an identifying FACET.

As an example, to show the total number of completed tasks by executor ID, use the query SELECT latest(app.executor.completedTasks) FROM Metric FACET sparkAppName, sparkAppExecutorId and not SELECT count(app.executor.completedTasks) FROM Metric FACET sparkAppName, sparkAppExecutorId. Using count() on a gauge metric accesses the count field of the metric which in this case is always 1 even though the metric being represented is a counter. The counter value is actually in the latest field of the metric. Further, using the metric without faceting by the executor ID will only return the latest app.executor.completedTasks metric in the selected time window and ignore any other instances of that metric in the same time window.

The following attributes are included on all Spark application metrics.

The following metrics are included for each Spark application.

The following metrics are included for each Spark application executor. Metrics are scoped to a given executor using the sparkAppExecutorId attribute.

The following attributes are included on all Spark application executor metrics.

The following metrics are included for each Spark job in an application. Metrics are scoped to a given job and status using the sparkAppJobId and sparkAppJobStatus attributes.

The following attributes are included on all Spark application job metrics.

The following metrics are included for each Spark stage in an application. Metrics are scoped to a given stage and status using the sparkAppStageId, sparkAppStageAttemptId, sparkAppStageName, and sparkAppStageStatus attributes.

NOTE: The metrics in this section are not documented in the Apache Spark monitoring ReST API documentation. The descriptions provided below were deduced via source code analysis and are not determinate.

The following attributes are included on all Spark application stage metrics.

The following metrics are included for each Spark task in an application. Metrics are scoped to a given stage and status using the sparkAppStageId, sparkAppStageAttemptId, sparkAppStageName, and sparkAppStageStatus attributes. They are also scoped to a given task and status using the sparkAppTaskId, sparkAppTaskAttempt, and sparkAppTaskStatus attributes and scoped to the executor of the task using the sparkAppTaskExecutorId attribute.

NOTE: Some of the shuffle read metric descriptions below are sourced from the file ShuffleReadMetrics.scala.

The following attributes are included on all Spark application stage task metrics.

The following metrics are included for each Spark RDD in an application. Metrics are scoped to a given RDD using the sparkAppRDDId and sparkAppRDDName attributes.

NOTE: The metrics in this section are not documented in the Apache Spark monitoring ReST API documentation. The descriptions provided below were deduced via source code analysis and are not determinate.

The following attributes are included on all Spark application RDD metrics.

Most of the Spark metrics recorded include at least one of the following attributes that indicate the status of the job, stage, or task for the metric.

• sparkAppJobStatus

  Spark job status. One of the following.

  • running - job is executing
  • lost - job status is unknown
  • succeeded - job completed successfully
  • failed - job failed

• sparkAppStageStatus

  Spark stage status. One of the following.

  • active - stage is executing
  • complete - stage completed successfully
  • skipped - stage was skipped because it did not need to be recomputed
  • failed - stage failed
  • pending- stage is waiting to be executed

• sparkAppTaskStatus

  Spark task status. One of the following.

  • active - task is executing
  • complete - task completed successfully
  • skipped - task was skipped because the stage was skipped
  • failed - task failed
  • killed - task was explicitly killed

On each run, for each Spark application, the integration records the following counter metrics.

• app.jobs

  The number of Spark jobs for an application by job status. This metric will always include the sparkAppJobStatus attribute, which indicates which job status the metric applies to (for instance, if the metric value is 5 and the sparkAppJobStatus attribute is running, it means that there are 5 running jobs).

  Use the sparkAppId or sparkAppName to target a specific application or to group the values by application.

• app.stages

  The number of Spark stages for an application by stage status. This metric will always include the sparkAppStageStatus attribute, which indicates which stage status the metric applies to (for instance, if the metric value is 5 and the sparkAppStageStatus attribute is complete, it means that there are 5 completed stages).

  Use the sparkAppId or sparkAppName attributes to target a specific application or to group the values by application.

• app.job.stages

  The number of Spark stages for a Spark job by stage status. This metric will always include the sparkAppStageStatus and sparkAppJobStatus attributes, which indicate which stage status and job status the metric applies to, respectively (for instance, if the metric value is 2 and the sparkAppStageStatus attribute is complete and the sparkAppJobStatus is running, it means that the job is running and 2 stages have completed).

  Use the sparkAppId or sparkAppName attributes to target a specific application or to group the values by application and the sparkAppJobId attribute to target a specific job or to group the values by job.

• app.job.tasks

  The number of Spark tasks for a Spark job by task status. This metric will always include the sparkAppTaskStatus and sparkAppJobStatus attributes, which indicate which task status and job status the metric applies to, respectively (for instance, if the metric value is 4 and the sparkAppTaskStatus attribute is complete and the sparkAppJobStatus is running, it means that the job is running and 4 tasks have completed).

  Use the sparkAppId or sparkAppName attributes to target a specific application or to group the values by application and the sparkAppJobId attribute to target a specific job or to group the values by job.

• app.stage.tasks

  The number of Spark tasks for a Spark stage by task status. This metric will always include the sparkAppTaskStatus and sparkAppStageStatus attributes, which indicate which task status and stage status the metric applies to, respectively (for instance, if the metric value is 4 and the sparkAppTaskStatus attribute is complete and the sparkAppStageStatus is active, it means that the stage is running and 4 tasks have completed).

  Use the sparkAppId or sparkAppName attributes to target a specific application or to group the values by application and the sparkAppStageId or sparkAppStageName attribute to target a specific stage or to group the values by stage.

In general, only the latest() aggregator function should be used when visualizing or alerting using these counters.

When the Databricks Integration is deployed on the driver node of a Databricks cluster, all Spark metrics collected from Spark running on Databricks compute will have the following attributes.

Additionally, the integration will attempt to map Spark job, stage, and task metrics to either a Databricks Job task run or a Databricks Pipeline (and in some cases a specific update flow). This is done by examining the jobGroup field that is returned for Spark jobs on the Spark ReST API.

By default, the value of the jobGroup field is a string with a specific format that includes the Databricks job ID and task run ID for Spark job, stage, and task metrics associated with a Databricks job run, and includes the Databricks pipeline ID (and in some cases the pipeline update ID and pipeline update flow ID) for Spark job, stage, and task metrics associated with a Databricks pipeline update.

When collecting Spark job, stage, and task metrics, the Databricks integration will examine the value of the jobGroup field. If the value follows one of the recognized formats, the integration will parse the IDs out of the field and include the IDs as attributes on each Spark job, stage, and task metric as follows.

• For Spark job, stage, and task metrics associated with a Databricks job task run, the following attributes are added.

  Name	Data Type	Description
  databricksJobId	number	The job ID for the associated Databricks job run
  databricksJobRunTaskRunId	number	The task run ID for the associated Databricks job run (only included if includeJobRunTaskRunId is set to true)

• For Spark job, stage, and task metrics associated with a Databricks pipeline update, the following attributes are added.

  Name	Data Type	Description
  databricksPipelineId	string	The pipeline ID for the associated Databricks pipeline
  databricksPipelineUpdateId	string	The pipeline update ID for the associated Databricks pipeline update (only included if the pipeline update ID is present in the jobGroup field and includePipelineUpdateId is set to true)
  databricksPipelineFlowId	string	The pipeline update flow ID for the flow in the associated Databricks pipeline update (only included if the pipeline update flow ID is present in the jobGroup field and databricksPipelindFlowId is set to true)

If the value of the jobGroup field does not follow one of the recognized formats (for example, if the jobGroup has been customized using the setJobGroup function), the field value will not be parsed and none of the additional attributes will be available on the associated Spark job, stage and task metrics. Only the workspace related attributes will be available.

All examples below assume the spark metricPrefix is spark..

Current number of jobs by application name and job status

FROM Metric
SELECT latest(spark.app.jobs)
FACET sparkAppName, sparkAppJobStatus

Current number of stages by application name and stage status

FROM Metric
SELECT latest(spark.app.stages)
FACET sparkAppName, sparkAppStageStatus

Current number of stages by application name, job ID, and stage status

FROM Metric
SELECT latest(spark.app.job.stages)
FACET sparkAppName, sparkAppJobId, sparkAppStageStatus

Current number of tasks by application name, stage ID, and task status

FROM Metric
SELECT latest(spark.app.stage.tasks)
FACET sparkAppName, sparkAppStageId, sparkAppTaskStatus
LIMIT MAX

Current number of running jobs by application name

FROM Metric
SELECT latest(spark.app.jobs)
WHERE sparkAppJobStatus = 'running'
FACET sparkAppName

Number of completed tasks by application name and job ID

FROM Metric
SELECT latest(spark.app.job.tasks)
WHERE sparkAppTaskStatus = 'complete'
FACET sparkAppName, sparkAppJobId
LIMIT MAX

Number of failed tasks by application name and job ID

FROM Metric
SELECT latest(spark.app.job.tasks)
WHERE sparkAppTaskStatus = 'failed'
FACET sparkAppName, sparkAppJobId
LIMIT MAX

Job duration by application name, job ID, and job status

FROM Metric
SELECT latest(spark.app.job.duration) / 1000 AS Duration
FACET sparkAppName, sparkAppJobId, sparkAppJobStatus
LIMIT MAX

Stage duration by application name, stage ID, and stage name

FROM Metric
SELECT latest(spark.app.stage.duration) / 1000 AS Duration
FACET sparkAppName, sparkAppStageId, sparkAppStageName
LIMIT MAX

Task duration by application name, stage ID, stage name, and task ID

FROM Metric
SELECT latest(spark.app.stage.task.executorRunTime) / 1000 AS Duration
FACET sparkAppName, sparkAppStageId, sparkAppStageName, sparkAppTaskId
LIMIT MAX

Total elapsed stage executor run time (in seconds) by application name, stage ID, and stage name

FROM Metric
SELECT latest(spark.app.stage.executor.runTime) / 1000 AS Duration
FACET sparkAppName, sparkAppStageId, sparkAppStageName
LIMIT MAX

Total elapsed stage JVM GC time (in seconds) by application name, stage ID, and stage name

FROM Metric
SELECT latest(spark.app.stage.jvmGcTime) / 1000 AS Duration
FACET sparkAppName, sparkAppStageId, sparkAppStageName
LIMIT MAX

Average memory used by application name and executor ID over time

FROM Metric
SELECT average(spark.app.executor.memoryUsed)
WHERE spark.app.executor.memoryUsed IS NOT NULL
FACET sparkAppName, sparkAppExecutorId
TIMESERIES

Number of executors (active and dead) by application name

FROM Metric
SELECT uniqueCount(sparkAppExecutorId)
WHERE metricName = 'spark.app.executor.maxMemory'
FACET sparkAppName

Total number of partitions by application name and RDD name

FROM Metric
SELECT latest(spark.app.storage.rdd.partitions)
FACET sparkAppName, sparkAppRDDName

Average RDD memory used by application name and RDD name

FROM Metric
SELECT average(spark.app.storage.rdd.memory.used)
WHERE spark.app.storage.rdd.memory.used IS NOT NULL
FACET sparkAppName, sparkAppRDDId

Average RDD partition memory used by application name, RDD ID, and RDD block name

FROM Metric
SELECT average(spark.app.storage.rdd.partition.memory.used)
WHERE spark.app.storage.rdd.partition.memory.used IS NOT NULL
FACET sparkAppName, sparkAppRDDId, sparkAppRddPartitionBlockName

A sample dashboard is included that shows examples of the types of Apache Spark information that can be displayed and the NRQL statements to use to visualize the data.

The Databricks Integration can collect Databricks consumption and cost related data from the Databricks system tables. This data can be used to show Databricks DBU consumption metrics and estimated Databricks costs directly within New Relic.

This feature is enabled by setting the Databricks usage enabled flag to true in the integration configuration. When enabled, the Databricks collector will collect consumption and cost related data once a day at the time specified in the runTime configuration parameter. The following information is collected on each run.

• Billable usage records from the system.billing.usage table
• List pricing records from the system.billing.list_prices table
• List cost per job run of jobs run on jobs compute and serverless compute
• List cost per job of jobs run on jobs compute and serverless compute
• List cost of failed job runs for jobs with frequent failures run on jobs compute and serverless compute
• List cost of repair job runs for jobs with frequent repairs run on jobs compute and serverless compute

NOTE: Job cost data from jobs run on workspaces outside the region of the workspace containing the SQL Warehouse used to collect the consumption data (specified in the workspaceHost configuration parameter) will not be returned by the queries used by the Databricks integration to collect Job cost data.

In order for the Databricks Integration to collect consumption and cost related data from Databricks, there are several requirements.

1. The SQL warehouse ID of a SQL warehouse within the workspace associated with the configured workspace host must be specified. The Databricks SQL queries used to collect consumption and cost related data from the Databricks system tables. will be run against the specified SQL warehouse.
2. As of v2.6.0, the Databricks integration leverages data that is stored in the system.lakeflow.jobs table. As of 10/24/2024, this table is in public preview. To access this table, the lakeflow schema must be enabled in your system catalog. To enable the lakeflow schema, follow the instructions in the Databricks documentation to enable a system schema using the metastore ID of the Unity Catalog metastore attached to the workspace associated with the configured workspace host and the schema name lakeflow.

On each run, billable usage data is collected from the system.billing.usage table for the previous day. For each billable usage record, a corresponding record is created within New Relic as a New Relic event with the event type DatabricksUsage and the following attributes.

NOTE: Not every attribute is included in every event. For example, the cluster_* attributes are only included in events for usage records relevant to all-purpose or job related compute. Similarly, the warehouse_* attributes are only included in events for usage records relevant to SQL warehouse related compute.

NOTE: Descriptions below are sourced from the billable usage system table reference.

In addition, all compute resource tags, jobs tags, and budget policy tags applied to this usage are added as event attributes. For jobs referenced in the job_id attribute of JOBS usage records, custom job tags are also included if the job was run within a workspace in the same cloud region as the workspace containing the SQL warehouse used to collect the consumption data (specified in the workspaceHost configuration parameter).

On each run, list pricing data is collected from the system.billing.list_prices table and used to populate a New Relic lookup table named DatabricksListPrices. The entire lookup table is updated on each run. For each pricing record, this table will contain a corresponding row with the following columns.

NOTE: Descriptions below are sourced from the pricing system table reference.

Note that the list_price field contains a single price suitable for simple long-term estimates. It does not reflect any promotional pricing or custom price plans.

On each run, the Databricks collector runs a set of queries that leverage data in the system.billing.usage table, the system.billing.list_prices table, and the system.lakeflow.jobs table to collect job cost related data for the previous day. The set of queries that are run collect the following data.

• List cost per job run of jobs run on jobs compute and serverless compute
• List cost per job of jobs run on jobs compute and serverless compute
• List cost of failed job runs for jobs with frequent failures run on jobs compute and serverless compute
• List cost of repaired job runs for jobs with frequent repairs run on jobs compute and serverless compute

By default, the Databricks integration runs each query on every run. This behavior can be configured using the optionalQueries configuration parameter to selectively enable or disable queries by query ID.

For each query that is run, a New Relic event with the event type DatabricksJobCost is created for each result row with one attribute for each column of data. In addition, each event includes a query_id attribute that contains the unique ID of the query that produced the data stored in the event. This ID can be used in NRQL queries to scope the query to the appropriate data set.

See the following sub-sections for more details on the data sets collected by each query.

NOTE: Please see the note at the end of the Consumption & Cost Data section regarding job cost data and the last requirement in the Consumption and Cost Requirements section for important considerations for collecting job cost data.

Job cost data on the list cost per job run is collected using the query with the query id jobs_cost_list_cost_per_job_run. This query produces DatabricksJobCost events with the following attributes.

Job cost data on the list cost per job is collected using the query with the query id jobs_cost_list_cost_per_job. This query produces DatabricksJobCost events with the following attributes.

Job cost data on the list cost of failed job runs for jobs with frequent failures is collected using the query with the query id jobs_cost_frequent_failures. This query produces DatabricksJobCost events with the following attributes.

Job cost data on the list cost of repaired job runs for jobs with frequent repairs is collected using the query with the query id jobs_cost_most_retries. This query produces DatabricksJobCost events with the following attributes.

The Databricks Integration can collect telemetry about Databricks Job runs, such as job run durations, task run durations, the current state of job and task runs, if a job or a task is a retry, and the number of times a task was retried. This feature is enabled by default and can be enabled or disabled using the Databricks jobs enabled flag in the integration configuration.

NOTE: Some of the text below is sourced from the Databricks SDK Go module documentation.

On each run, the integration uses the Databricks ReST API to retrieve job run data. By default, the Databricks ReST API endpoint for listing job runs returns a paginated list of all historical job runs sorted in descending order by start time. On systems with many jobs, there may be a large number of job runs to retrieve and process, impacting the performance of the collector. To account for this, the integration provides the startOffset configuration parameter. This parameter is used to tune the performance of the integration by constraining the start time of job runs to match to be greater than a particular time in the past calculated as an offset from the current time and thus, limiting the number of job runs to return.

The effect of this behavior is that only job runs which have a start time at or after the calculated start time will be returned on the API call. For example, using the default startOffset (86400 seconds or 24 hours), only job runs which started within the last 24 hours will be returned. This means that jobs that started more than 24 hours ago will not be returned even if some of those jobs are not yet in a TERMINATED state. Therefore, it is important to carefully select a value for the startOffset parameter that will account for long-running job runs without degrading the performance of the integration.

Job run data is sent to New Relic as dimensional metrics. The following metrics and attributes (dimensions) are provided.

Databricks job and task runs can be in one of 6 states.

• BLOCKED - run is blocked on an upstream dependency
• PENDING - run is waiting to be executed while the cluster and execution context are being prepared
• QUEUED - run is queued due to concurrency limits
• RUNNING - run is executing
• TERMINATING - run has completed, and the cluster and execution context are being cleaned up
• TERMINATED - run has completed

The job and run task states are recorded in the databricksJobRunState and databricksJobRunTaskState, respectively. The databricksJobRunState is on every job run metric including the task related metrics. The databricksJobRunTaskState is only on job run metrics related to tasks, e.g. job.run.task.duration.

On each run, the integration records the number of job and task runs in each state (with the exception of runs in the TERMINATED state (see below)), in the metrics job.runs and job.tasks, respectively, using the attributes databricksJobRunState and databricksJobRunTaskState, to indicate the state. In general, only the latest() aggregator function should be used when visualizing or alerting on counts including these states. For example, to display the number of jobs by state, use the following NRQL statement.

FROM Metric
SELECT latest(databricks.job.runs)
FACET databricksJobRunState

The count of job and task runs in the TERMINATED state are also recorded but only include job and task runs that have terminated since the last run of the integration. This is done to avoid counting terminated job and task runs more than once, making it straightforward to use aggregator functions to visualize or alert on values such as "number of job runs completed per time period", "average duration of job runs", and "average duration job runs spent in the queue". For example, to show the average time job runs spend in the queue, grouped by job name, use the following NRQL statement.

FROM Metric
SELECT average(databricks.job.run.duration.queue / 1000)
WHERE databricksJobRunState = 'TERMINATED'
FACET databricksJobRunName
LIMIT MAX

NOTE: Make sure to use the condition databricksJobRunState = 'TERMINATED' or databricksJobRunTaskState = 'TERMINATED' when visualizing or alerting on values using these aggregator functions.

The Databricks integration stores job and task run durations in the metrics job.run.duration and job.run.task.duration, respectively. The values stored in these metrics are calculated differently, depending on the state of the job or task, as follows.

• While a job or task run is running (not BLOCKED or TERMINATED), the run duration stored in the respective metrics is the "wall clock" time of the job or task run, calculated as the current time when the metric is collected by the integration minus the start_time of the job or task run as returned from the Databricks ReST API. This duration is inclusive of any time spent in the queue and any setup time, execution time, and cleanup time that has been spent up to the time the duration is calculated but does not include any time the job or task run was blocked.

  This duration is also cumulative, meaning that while the job run is running, the value calculated each time the integration runs will include the entire "wall clock" time since the reported start_time. It is essentially a cumulativeCount metric but stored as a gauge. Within NRQL statements, the latest() function can be used to get the most recent duration. For example, to show the most recent duration of job runs that are running grouped by job name, use the following NRQL (assuming the job run metricPrefix is databricks.).

  FROM Metric
  SELECT latest(databricks.job.run.duration)
  WHERE databricksJobRunState != 'TERMINATED'
  FACET databricksJobRunName

• Once a job or task run has been terminated, the run duration stored in the respective metrics are not calculated but instead come directly from the values returned in the listruns API (also see the SDK documentation for BaseRun and RunTask). For both job and task runs, the run durations are determined as follows.

  • For job runs of multi-task jobs, the job.run.duration for a job run is set to the run_duration field of the corresponding job run item in the runs field returned from the listruns API.
  • For job runs of single-task jobs, the job.run.duration for a job run is set to the sum of the setup_duration, the execution_duration and the cleanup_duration fields of the corresponding job run item in the runs field returned from the listruns API.
  • For task runs, the job.run.task.duration for a task run is set to the sum of the setup_duration, the execution_duration and the cleanup_duration fields of the corresponding task run item in the tasks field of the corresponding job run item in the runs field returned from the listruns API.

In addition to the job.run.duration and job.run.task.duration metrics, once a job or task has terminated, the job.run.duration.queue and job.run.task.duration.queue metrics are set to the queue_duration field of the corresponding job run item in the runs field returned from the listruns API for job runs, and the queue_duration field of the corresponding task run item in the tasks field of the corresponding job run item in the runs field from the listruns API for task runs.

Finally, for job runs and task runs of single-task jobs, the following metrics are set.

• The job.run.duration.setup and job.run.task.duration.setup metrics are set to the setup_duration field of the corresponding job run item in the runs field returned from the listruns API for job runs, and the setup_duration field of the corresponding task run item in the tasks field of the corresponding job run item in the runs field from the listruns API for task runs.
• The job.run.duration.execution and job.run.task.duration.execution are set to the execution_duration field of the corresponding job run item in the runs field returned from the listruns API for job runs, and the execution_duration field of the corresponding task run item in the tasks field of the corresponding job run item in the runs field from the listruns API for task runs.
• The job.run.duration.cleanup and job.run.task.duration.cleanup are set to the cleanup_duration field of the corresponding job run item in the runs field returned from the listruns API for job runs, and the cleanup_duration field of the corresponding task run item in the tasks field of the corresponding job run item in the runs field from the listruns API for task runs.

All examples below assume the job run metricPrefix is databricks..

Current job run counts by state

FROM Metric
SELECT latest(databricks.job.runs)
FACET databricksJobRunState

Current task run counts by state

FROM Metric
SELECT latest(databricks.job.tasks)
FACET databricksJobRunTaskState

Job run durations by job name over time

FROM Metric
SELECT latest(databricks.job.run.duration)
FACET databricksJobRunName
TIMESERIES

Average job run duration by job name

FROM Metric
SELECT average(databricks.job.run.duration)
WHERE databricksJobRunState = 'TERMINATED'
FACET databricksJobRunName

NOTE: Make sure to use the condition databricksJobRunState = 'TERMINATED' or databricksJobRunTaskState = 'TERMINATED' when visualizing or alerting on values using aggregator functions other than latest.

Average task run queued duration by task name over time

FROM Metric
SELECT average(databricks.job.run.task.duration.queue)
WHERE databricksJobRunTaskState = 'TERMINATED'
FACET databricksJobRunTaskName
TIMESERIES

NOTE: Make sure to use the condition databricksJobRunState = 'TERMINATED' or databricksJobRunTaskState = 'TERMINATED' when visualizing or alerting on values using aggregator functions other than latest.

A sample dashboard is included that shows examples of the types of job run information that can be displayed and the NRQL statements to use to visualize the data.

The Databricks Integration can collect telemetry about Databricks Delta Live Tables Pipeline updates, such as flow durations, flow data quality metrics, and update durations. This feature is enabled by default and can be enabled or disabled using the Databricks pipeline update metrics enabled flag in the integration configuration.

NOTE: Some of the text below is sourced from the Databricks Delta Live Tables pipeline event log schema documentation and the Databricks SDK Go module documentation.

On each run, the integration uses the Databricks ReST API to retrieve pipeline events for all Delta Live Tables Pipelines. By default, the Databricks ReST API endpoint for listing pipeline events returns a paginated list of all historical pipeline events sorted in descending order by timestamp. On systems with many pipelines or complex pipelines with many flows, there may be a large number of pipeline events to retrieve and process, impacting the performance of the collector. To account for this, the integration provides the startOffset configuration parameter. This parameter is used to tune the performance of the integration by constraining the timestamp of pipeline events to match to be greater than a particular time in the past calculated as an offset from the current time and thus, limiting the number of pipeline events to return.

The effect of this behavior is that only pipeline events which have a timestamp at or after the calculated start time will be returned on the API call. For example, using the default startOffset (86400 seconds or 24 hours), only pipeline events which occurred within the last 24 hours will be returned. This means that pipeline events that occurred more than 24 hours ago will not be returned even if the associated pipeline update is not yet complete. This can lead to missing flow or update metrics. Therefore, it is important to carefully select a value for the startOffset parameter that will account for long-running pipeline updates without degrading the performance of the integration.

The integration uses the list pipeline events endpoint of the Databricks ReST API to retrieve pipeline log events. There can be a slight lag time between when a pipeline log event is emitted and when it is available in the API. The intervalOffset configuration parameter can be used to delay the processing of pipeline log events in order to account for this lag. The default value for the intervalOffset configuration parameter is 5 seconds.

To help illustrate, here is the processing behavior for the default settings (60 second collection interval and 5 second interval offset):

• on the first minute, the integration will process all pipeline log events from 60 seconds ago until 55 seconds ago
• on the second interval, the integration will process all pipeline log events from 55 seconds ago until 1:55
• on the third interval, the integration will process all pipeline log events from 1:55 until 2:55
• and so on

Pipeline update metric data is sent to New Relic as dimensional metrics. The following metrics and attributes (dimensions) are provided.

Databricks pipelines can be in one of nine states.

• DELETED
• DEPLOYING
• FAILED
• IDLE
• RECOVERING
• RESETTING
• RUNNING
• STARTING
• STOPPING

The pipeline state is recorded in the databricksPipelineState attribute on the pipeline.pipelines metric only.

Databricks pipeline updates can have one of eleven statuses.

• CANCELED
• COMPLETED
• CREATED
• FAILED
• INITIALIZING
• QUEUED
• RESETTING
• RUNNING
• SETTING_UP_TABLES
• STOPPING
• WAITING_FOR_RESOURCES

The pipeline update status is recorded in the databricksPipelineUpdateStatus attribute on the pipeline.updates metric and the pipeline.update.* metrics.

The statuses CANCELED, COMPLETED, and FAILED are considered update termination statuses.

Databricks pipeline flows can have one of ten statuses.

• COMPLETED
• EXCLUDED
• FAILED
• IDLE
• PLANNING
• QUEUED
• RUNNING
• SKIPPED
• STARTING
• STOPPED

The pipeline flow status is recorded in the databricksPipelineFlowStatus attribute on the pipeline.flows metric and the pipeline.flow.* metrics.

The statuses COMPLETED, EXCLUDED, FAILED, SKIPPED, and STOPPED are considered flow termination statuses.

On each run, the integration records the following counts.

• The count of all pipelines by pipeline state is recorded in the metric pipeline.pipelines using the attribute databricksPipelineState to indicate the pipeline state.

• The count of all pipeline updates by update status is recorded in the metric pipeline.updates using the attribute databricksPipelineUpdateStatus to indicate the update status.

  Pipeline updates with a termination update status (CANCELED, COMPLETED, FAILED) are only counted if the corresponding event occurred since the last run of the integration. Otherwise, these updates are ignored. This is done to avoid counting terminated updates more than once.

  Pipeline updates with a non-termination update status are always counted.

• The count of all pipeline flows by flow status is recorded in the metric pipeline.flows using the attribute databricksPipelineFlowStatus to indicate the flow status.

  Pipeline flows with a termination flow status (COMPLETED, EXCLUDED, FAILED, SKIPPED, STOPPED) are only counted if the corresponding event occurred since the last run of the integration. Otherwise, these flows are ignored. This is done to avoid counting terminated flows more than once.

  Pipeline flows with a non-termination flow status are always counted.

The Databricks integration records several update and flow durations. The different durations and the ways in which they are calculated are as follows.

• pipeline.update.duration

  The pipeline.update.duration metric represents the total duration of the update from the time it is created until the time it terminates. After the update is created but before it terminates, the duration recorded in this metric is the "wall clock" time, calculated as the current time when the metric is recorded by the integration minus the timestamp of the pipeline log event where the update was created. Once the update terminates, the duration recorded in this metric is calculated by subtracting the timestamp of the pipeline log event where the update was created from the timestamp of the pipeline log event where the update reached the termination status.

• pipeline.update.duration.wait

  The pipeline.update.duration.wait metric represents the amount of time the update spent waiting for resources (that is, while it has the WAITING_FOR_RESOURCES update status). While the update is waiting for resources, the duration recorded in this metric is the "wall clock" time, calculated as the current time when the metric is recorded by the integration minus the timestamp of the pipeline log event where the update reached the WAITING_FOR_RESOURCES status. Once the update reaches the INITIALIZING status or terminates, the duration recorded in this metric is calculated by subtracting the timestamp of the pipeline log event where the update reached the WAITING_FOR_RESOURCES status from the timestamp of the pipeline log event where the update reached the INITIALIZING status or terminated.

• pipeline.update.duration.runTime

  The pipeline.update.duration.runTime metric represents the amount of time the update spent actually running, defined as the time the update reached the INITIALIZING update status until the time the update terminates. After the update has reached the INITIALIZING status but before it terminates, the duration recorded in this metric is the "wall clock" time, calculated as the current time when the metric is recorded by the integration minus the timestamp of the pipeline log event where the update reached the INITIALIZING status. Once the update terminates, the duration recorded in this metric is calculated by subtracting the timestamp of the pipeline log event where the update reached the INITIALIZING status from the timestamp of the pipeline log event where the update terminated.

• pipeline.flow.duration

  The pipeline.flow.duration metric represents the total duration of the flow from the time it starts until the time it terminates. After the flow starts but before it terminates, the duration recorded in this metric is the "wall clock" time, calculated as the current time when the metric is recorded by the integration minus the timestamp of the pipeline log event where the flow terminated. Once the flow terminates, the duration recorded in this metric is calculated by subtracting the timestamp of the pipeline log event where the flow started from the timestamp of the pipeline log event where the flow terminated.

• pipeline.flow.duration.queue

  The pipeline.flow.duration.queue metric represents the time the flow spent in the queue (that is, while it has the QUEUED flow status). While the update in the queue and before it terminates or reaches the PLANNING or STARTING status, the duration recorded in this metric is the "wall clock" time, calculated as the current time when the metric is recorded by the integration minus the timestamp of the pipeline log event where the flow reached the QUEUED status. Once the flow terminates or reaches either the PLANNING status or the STARTING status, the duration recorded in this metric is calculated by subtracting the timestamp of the pipeline log event where the flow reached the QUEUED status from the timestamp of the log event where the flow terminated or reached either the PLANNING status or the STARTING status.

• pipeline.flow.duration.plan

  The pipeline.flow.duration.plan metric represents the time spent to plan the flow (that is, while it has the PLANNING flow status). While the update has the PLANNING status and before it terminates or reaches the STARTING status, the duration recorded in this metric is the "wall clock" time, calculated as the current time when the metric is recorded by the integration minus the timestamp of the pipeline log event where the flow terminated or reached the PLANNING status. Once the terminates or reaches the STARTING status, the duration recorded in this metric is calculated by subtracting the timestamp of the pipeline log event where the flow reached the PLANNING status from the timestamp of the log event where the flow terminated or reached the STARTING status.

NOTE: Wall clock durations are cumulative, meaning that until the update or flow reaches the status at which the true duration is calculated, the value of each successive metric increases monotonically. It is essentially a cumulativeCount metric but stored as a gauge. Within NRQL statements, the latest() function can be used to get the most recent duration. For example, to show the most recent duration of flows that are running grouped by flow name, use the following NRQL (assuming the pipeline update metricPrefix is databricks.).

FROM Metric
SELECT latest(databricks.pipeline.flow.duration)
WHERE databricksPipelineFlowStatus = `RUNNING`
FACET databricksPipelineFlowName

All examples below assume the pipeline update metricPrefix is databricks..

Current update counts by state

FROM Metric
SELECT latest(databricks.pipeline.pipelines) AS 'Pipelines'
FACET databricksPipelineState

Current update counts by status

FROM Metric
SELECT latest(databricks.pipeline.updates) AS 'Updates'
FACET databricksPipelineUpdateStatus

Update durations by pipeline name and update ID over time

FROM Metric
SELECT latest(databricks.pipeline.update.duration)
FACET databricksPipelineName, substring(databricksPipelineUpdateId, 0, 6)
TIMESERIES

Recent updates

FROM Metric
SELECT databricksPipelineName AS Pipeline, substring(databricksPipelineUpdateId, 0, 6) AS Update, databricksPipelineUpdateStatus as Status, getField(databricks.pipeline.update.duration, latest) / 1000 AS Duration
WHERE databricks.pipeline.update.duration IS NOT NULL
 AND databricksPipelineUpdateStatus IN ('COMPLETED', 'CANCELED', 'FAILED')
LIMIT MAX

Average update duration by pipeline name and update status

FROM Metric
SELECT average(databricks.pipeline.update.duration) / 1000 AS 'seconds'
WHERE databricksPipelineUpdateStatus IN ('COMPLETED', 'CANCELED', 'FAILED')
FACET databricksPipelineName, databricksPipelineUpdateStatus
TIMESERIES

NOTE: Make sure to use the condition databricksPipelineUpdateStatus IN ('COMPLETED', 'CANCELED', 'FAILED') when visualizing or alerting on update duration using aggregator functions other than latest.

Recent flows

FROM Metric
SELECT databricksPipelineName AS Pipeline, substring(databricksPipelineUpdateId, 0, 6) AS Update, databricksPipelineFlowName AS Flow, databricksPipelineFlowStatus AS Status, getField(databricks.pipeline.flow.duration, latest) / 1000 AS Duration
WHERE databricks.pipeline.flow.duration IS NOT NULL
 AND databricksPipelineFlowStatus IN ('COMPLETED', 'STOPPED', 'SKIPPED', 'FAILED', 'EXCLUDED')
LIMIT MAX

Average flow backlog bytes

FROM Metric
SELECT average(databricks.pipeline.flow.backlogBytes)
FACET databricksPipelineName, databricksPipelineFlowName

Flow expectation records passed

FROM Metric
SELECT databricksPipelineName AS Pipeline, substring(databricksPipelineUpdateId, 0, 6) AS Update, databricksPipelineFlowName AS Flow, databricksPipelineFlowExpectationName AS Expectation, getField(databricks.pipeline.flow.expectation.recordsPassed, latest) AS Records
WHERE databricks.pipeline.flow.expectation.recordsPassed IS NOT NULL
LIMIT MAX

A sample dashboard is included that shows examples of the types of pipeline update information that can be displayed and the NRQL statements to use to visualize the data.

The Databricks Integration can collect Databricks Delta Live Tables Pipeline event logs for all Databricks Delta Live Tables Pipelines defined in a workspace. Databricks Delta Live Tables Pipeline event log entries for every pipeline update are collected and sent to New Relic Logs.

NOTE: Some of the text below is sourced from the Databricks Delta Live Tables pipeline event log schema documentation and the Databricks SDK Go module documentation.

Databricks Delta Live Tables Pipeline event logs are sent to New Relic as New Relic Log data. For each Databricks Delta Live Tables Pipeline event log entry, the fields from the event log entry are mapped to attributes on the corresponding New Relic log entry as follows.

NOTE: Due to the way in which the Databricks ReST API returns Databricks Delta Live Tables Pipeline event log entries (in descending order by timestamp), later event log entries may sometimes be visible in the New Relic Logs UI before older event log entries are visible. This does not affect the temporal ordering of the log entries. Once older log entries become visible (usually 30-60s longer than the value of the interval configuration parameter), they are properly ordered by timestamp in the New Relic Logs UI.

Additionally, if the error field is set on the pipeline event log entry, indicating an error was captured by the event, the following attributes are added to the New Relic log entry.

The New Relic Infrastructure agent can be used to collect infrastructure data like CPU and memory usage from the nodes in a Databricks cluster as well as the Spark driver and executor logs, the Spark driver event log, and the logs from all init scripts on the driver and worker nodes.

The New Relic Infrastructure agent will be automatically installed on the driver and worker nodes of the cluster as part of the process of deploying the integration on the driver node of a cluster if the value of the NEW_RELIC_INFRASTRUCTURE_ENABLED environment variable is set to true. In addition, if the value of the environment variable NEW_RELIC_INFRASTRUCTURE_LOGS_ENABLED is set to true, the Spark driver and executor logs, the Spark driver event log, and the logs from all init scripts on the driver and worker nodes will be collected and forwarded to New Relic Logs.

When installed, the New Relic infrastructure agent collects all default infrastructure monitoring data for the driver and worker nodes. Additionally, a host entity will show up for each node in the cluster.

When log forwarding is configured, the following logs are collected.

• Spark driver standard out and standard error logs
• Spark driver log4j log
• Spark event log
• Standard out and standard error logs for all init scripts on the driver node
• Spark application executor standard out and standard error logs
• Standard out and standard error logs for all init scripts on the worker nodes

The following attributes are included on all infrastructure data, all Log events collected by the infrastructure agent, and on the host entities for all nodes in the cluster.

The databricksLogType attribute can have one of the following values. These values can be used to determine the log which generated the Log event.

• driver-stdout

  Indicates the Log event originated from the driver standard out log.

• driver-stderr

  Indicates the Log event originated from the driver standard error log.

• driver-log4j

  Indicates the Log event originated from the driver log4j log.

• spark-eventlog

  Indicates the Log event originated from the Spark event log.

• driver-init-script-stdout

  Indicates the Log event originated from the standard out log of an init script on the driver node. The specific init script can be determined by examining the filePath attribute.

• driver-init-script-stderr

  Indicates the Log event originated from the standard error log of an init script on the driver node. The specific init script can be determined by examining the filePath attribute.

• executor-stdout

  Indicates the Log event originated from the standard out log of a Spark application executor. The specific Spark application can be determined by examining the filePath attribute.

• executor-stderr

  Indicates the Log event originated from the standard error log of a Spark application executor. The specific Spark application can be determined by examining the filePath attribute.

• worker-init-script-stdout

  Indicates the Log event originated from the standard out log of an init script on a worker node. The specific init script can be determined by examining the filePath attribute. The specific worker node can be determined by examining the hostname or fullHostname attributes.

• worker-init-script-stderr

  Indicates the Log event originated from the standard error log of an init script on a worker node. The specific init script can be determined by examining the filePath attribute. The specific worker node can be determined by examining the hostname or fullHostname attributes.

Driver node average CPU by cluster and entity name

FROM SystemSample
SELECT average(cpuPercent)
WHERE databricksIsDriverNode = 'true'
FACET databricksClusterName, entityName
TIMESERIES

Worker node average CPU by cluster and entity name

FROM SystemSample
SELECT average(cpuPercent)
WHERE databricksIsDriverNode = 'false'
FACET databricksClusterName, entityName
TIMESERIES

Spark driver log4j logs

WITH position(databricksWorkspaceHost, '.') AS firstDot
FROM Log
SELECT substring(databricksWorkspaceHost, 0, firstDot) AS 'Workspace', databricksClusterName AS 'Cluster', hostname AS Hostname, level AS Level, message AS Message
WHERE databricksClusterId IS NOT NULL
 AND databricksLogType = 'driver-log4j'
LIMIT 1000

Spark event logs

WITH position(databricksWorkspaceHost, '.') AS firstDot,
 capture(filePath, r'/databricks/driver/eventlogs/(?P<sparkContextId>[^/]+)/.*') AS sparkContextId
FROM Log
SELECT substring(databricksWorkspaceHost, 0, firstDot) AS 'Workspace', databricksClusterName AS 'Cluster', sparkContextId AS 'Spark Context ID', Event, `App Name`, `Job ID`, `Stage Info.Stage ID` AS 'Stage ID', `Task Info.Task ID` AS 'Task ID'
WHERE databricksClusterId IS NOT NULL
 AND databricksLogType = 'spark-eventlog'
LIMIT 1000

Spark application executor standard out logs

WITH position(databricksWorkspaceHost, '.') AS firstDot,
 capture(filePath, r'/databricks/spark/work/(?P<appName>[^/]+)/.*') AS appName
FROM Log
SELECT substring(databricksWorkspaceHost, 0, firstDot) AS 'Workspace', databricksClusterName AS 'Cluster', hostname AS Hostname, appName AS Application, message AS Message
WHERE databricksClusterId IS NOT NULL
 AND databricksLogType = 'executor-stdout'
LIMIT 1000

A sample dashboard is included that shows examples of visualizing cluster health metrics and logs and the NRQL statements to use to visualize the data.

The Databricks Integration can collect telemetry about queries executed in Databricks SQL warehouses and serverless compute, including execution times, query statuses, and query I/O metrics such as the number of bytes and rows read and the number of bytes spilled to disk and sent over the network. This feature is enabled by default and can be enabled or disabled using the Databricks query metrics enabled flag in the integration configuration.

NOTE:

• Unlike other metrics collected by the integration, query metrics are only recorded once a query has completed execution.
• When serverless compute is enabled, metrics for all SQL and Python queries executed on serverless compute for notebooks and jobs are returned by the list query history API used to collect query metrics.
• Some of the text below is sourced from the documentation on Databricks queries and the Databricks SDK Go module documentation.

Query metrics are only collected once when the query terminates, that is, when it completes successfully, when it fails, or when it is canceled. Furthermore, query metrics are only collected for queries that terminated since the last execution of the integration. This way, query metrics are only collected once.

Query metrics are collected using the query history endpoint. of the Databricks ReST API By default, this endpoint returns queries sorted in descending order by start time. This means that unless the time range used on the list query history API call includes the time when a query started, it will not be returned. Were the integration to only request the query history since the last execution of the integration, telemetry for queries that started prior to the last execution of the integration but completed since the last execution of the integration would not be recorded because they would not be returned.

To account for this, the integration provides the startOffset configuration parameter. This parameter is used to offset the start time of the time range used on the list query history API call. The effect of this behavior is that metrics for queries that completed since the last execution of the integration but started prior to the last execution of the integration (up to startOffset seconds ago) will be recorded.

For example, consider a query that started 3 minutes prior to the current execution of the integration but completed 10 seconds prior to the current execution of the integration. Using the default value of the interval configuration parameter, without a mechanism to effect the start time used to query the query history, the time range used on list query history API would be 60 seconds ago until 0 seconds ago. Because the list query history API returns queries according to the start time of the query, the example query would not be returned even though it completed since the last execution of the integration. Since the query is not returned, the query end time can not be checked against the time window and the query metrics will never be recorded.

In contrast, with the default startOffset (600 seconds or 10 minutes), the example query would be returned even though it started before the last execution of the integration. Since the query is returned, the integration will see that the query completed since the last execution of the integration, and will record the query metrics for the query.

NOTE: It is important to carefully select a value for the startOffset configuration parameter. Collecting query metrics with the list query history API is an expensive operation. A value should be chosen that accounts for as many long running queries as possible without negatively affecting the performance of the integration or the cost of executing the list query history API call.

Since query metrics are only collected if the query terminated since the last execution of the integration, if there is any lag setting the query end time, it is possible for some queries to be missed. For example, consider a query that ends one second prior to the current run time of the integration. If the end time of the query takes one second or more to be reflected in the API, the query end time will not be set during the current run of the integration and can not be checked. On the subsequent run of the integration, the query will be returned and it's end time will be set but the end time will fall just outside the last run time of the integration. As such, the metrics for the query will never be recorded.

To account for this, the integration provides the intervalOffset configuration parameter. This parameter is used to slide the time window that the query end time is checked against by some number of seconds, allowing the query end time additional time to be reflected in the API. The default value for the intervalOffset configuration parameter is 5 seconds.

Query metric data is sent to New Relic as events. The following event attributes are provided.

NOTE: Much of the text below is sourced from the file model.go.

Total completed queries

FROM DatabricksQuery
SELECT count(*) AS Queries
WHERE status = 'FINISHED'

Total failed queries

FROM DatabricksQuery
SELECT count(*) AS Queries
WHERE status = 'FAILED'

Rate of successful queries per second

FROM DatabricksQuery
SELECT rate(count(*), 1 second) AS Queries
WHERE status = 'FINISHED'
TIMESERIES

Number of failed queries by query text

FROM DatabricksQuery
SELECT count(*)
WHERE status = 'FAILED'
FACET query

Average bytes spilled to disk by query text

FROM DatabricksQuery
SELECT average(diskBytesSpilled)
WHERE diskBytesSpilled > 0
ORDER BY diskBytesSpilled DESC
FACET query

Total duration of all queries run by warehouse

FROM DatabricksQuery
SELECT sum(duration) / 1000 AS Duration
FACET warehouseName
TIMESERIES

Query history

WITH
 concat(workspaceUrl, '/sql/warehouses/', warehouseId, '/monitoring?queryId=', id) AS queryLink
FROM DatabricksQuery
SELECT
 status as Status,
 substring(query, 0, 100) as Query,
 startTime as Started,
 duration as Duration,
 userName as User,
 queryLink

Queries with data spilling

WITH
 concat(workspaceUrl, '/sql/warehouses/', warehouseId, '/monitoring?queryId=', id) AS queryLink
FROM DatabricksQuery
SELECT
 diskBytesSpilled,
 query,
 queryLink
WHERE diskBytesSpilled > 0
ORDER BY diskBytesSpilled DESC
LIMIT 100

A sample dashboard is included that shows examples of the types of query metric information that can be displayed and the NRQL statements to use to visualize the data.

While not strictly enforced, the basic preferred editor settings are set in the .editorconfig. Other than this, no style guidelines are currently imposed.

This project uses both go vet and staticcheck to perform static code analysis. These checks are run via precommit on all commits. Though this can be bypassed on local commit, both tasks are also run during the validate workflow and must have no errors in order to be merged.

Commit messages must follow the conventional commit format. Again, while this can be bypassed on local commit, it is strictly enforced in the validate workflow.

The basic commit message structure is as follows.

<type>[optional scope][!]: <description>

[optional body]

[optional footer(s)]

In addition to providing consistency, the commit message is used by svu during the release workflow. The presence and values of certain elements within the commit message affect auto-versioning. For example, the feat type will bump the minor version. Therefore, it is important to use the guidelines below and carefully consider the content of the commit message.

Please use one of the types below.

• feat (bumps minor version)
• fix (bumps patch version)
• chore
• build
• docs
• test

Any type can be followed by the ! character to indicate a breaking change. Additionally, any commit that has the text BREAKING CHANGE: in the footer will indicate a breaking change.

For local development, simply use go build and go run. For example,

go build cmd/databricks/databricks.go

Or

go run cmd/databricks/databricks.go

If you prefer, you can also use goreleaser with the --single-target option to build the binary for the local GOOS and GOARCH only.

goreleaser build --single-target

Releases are built and packaged using goreleaser. By default, a new release will be built automatically on any push to the main branch. For more details, review the .goreleaser.yaml and the goreleaser documentation.

The svu utility is used to generate the next tag value based on commit messages.

This project utilizes GitHub workflows to perform actions in response to certain GitHub events.

Sensitive data like credentials and API keys should never be specified directly in custom environment variables. Instead, it is recommended to create a secret using the Databricks CLI and reference the secret in the environment variable.

For example, the init script used to install the Databricks Integration when deploying the integration on the driver node of a cluster requires the New Relic license key to be specified in the environment variable named NEW_RELIC_LICENSE_KEY. The steps below show an example of how to create a secret for the New Relic license key and reference it in a custom environment variable.

1. Follow the steps to install or update the Databricks CLI.

2. Use the Databricks CLI to create a Databricks-backed secret scope with the name newrelic. For example,

   databricks secrets create-scope newrelic

   NOTE: Be sure to take note of the information in the referenced URL about the MANAGE scope permission and use the correct version of the command.

3. Use the Databricks CLI to create a secret for the license key in the new scope with the name licenseKey. For example,

   databricks secrets put-secret --json '{
      "scope": "newrelic",
      "key": "licenseKey",
      "string_value": "[YOUR_LICENSE_KEY]"
   }'

To set the custom environment variable named NEW_RELIC_LICENSE_KEY and reference the value from the secret, follow the steps to configure custom environment variables and add the following line after the last entry in the Environment variables field.

NEW_RELIC_LICENSE_KEY={{secrets/newrelic/licenseKey}}

New Relic has open-sourced this project. This project is provided AS-IS WITHOUT WARRANTY OR DEDICATED SUPPORT. Issues and contributions should be reported to the project here on GitHub.

We encourage you to bring your experiences and questions to the Explorers Hub where our community members collaborate on solutions and new ideas.

At New Relic we take your privacy and the security of your information seriously, and are committed to protecting your information. We must emphasize the importance of not sharing personal data in public forums, and ask all users to scrub logs and diagnostic information for sensitive information, whether personal, proprietary, or otherwise.

We define “Personal Data” as any information relating to an identified or identifiable individual, including, for example, your name, phone number, post code or zip code, Device ID, IP address, and email address.

For more information, review New Relic’s General Data Privacy Notice.

We encourage your contributions to improve this project! Keep in mind that when you submit your pull request, you'll need to sign the CLA via the click-through using CLA-Assistant. You only have to sign the CLA one time per project.

If you have any questions, or to execute our corporate CLA (which is required if your contribution is on behalf of a company), drop us an email at [email protected].

A note about vulnerabilities

As noted in our security policy, New Relic is committed to the privacy and security of our customers and their data. We believe that providing coordinated disclosure by security researchers and engaging with the security community are important means to achieve our security goals.

If you believe you have found a security vulnerability in this project or any of New Relic's products or websites, we welcome and greatly appreciate you reporting it to New Relic through HackerOne.

If you would like to contribute to this project, review these guidelines.

To all contributors, we thank you! Without your contribution, this project would not be what it is today.

The Databricks Integration project is licensed under the Apache 2.0 License.