Shutdown Timeout above Threshold
This policy specifies the minimum value which should be set for auto termination of a pool.
If the value is below the threshold, an incident will be generated.
Use this incident policy to generate incidents in order to minimize the amount of idling resources.
Databricks' pool configuration allows the user to specify an inactivity period in minutes after which the pool will be terminated automatically.
Databricks' allowed pool auto-termination interval must be between 1 and 10000 minutes.
Set the Idle Instance Auto Termination time to provide a buffer between when the instance is released from the cluster and when it’s dropped from the pool. Set this to a period that allows you to minimize cost while ensuring the availability of instances for scheduled jobs.
For example, job A is scheduled to run at 8:00 AM and takes 40 minutes to complete. Job B is scheduled to run at 9:00 AM and takes 30 minutes to complete. Set the Idle Instance Auto Termination value to 20 minutes to ensure that instances returned to the pool when job A completes are available when job B starts. Unless they are claimed by another cluster, those instances are terminated 20 minutes after job B ends.
Learn more:
Add Comment