Salesforce described measuring Data 360 Spark workloads running about eight million times a day and adjusting initial and maximum executor counts. One job’s initial allocation fell from 40 to four, while caps were reduced for groups with high idle ratios. Comparing four weeks before and after for the affected workloads, weekly costs fell 31% and execution starts rose 7%, but this was not a controlled experiment. Spark documentation notes that a separately specified executor count can override a smaller initial value, so changing one setting cannot guarantee the same savings.
Back to Tech
Salesforce reports lower costs after tuning Spark execution resources
Measure idle resources and initial allocations before reducing them by workload group to assess costs alongside processing stability.