Create Spark Helm Chart (latest version)

Hi, what I need is the following:

- I need a helm script (kubernetes) which installs the latest spark version as yarn cluster on kubernetes

- This setup should provide Zeppelin

- All of this has to work with sparks latest structured streaming framework and should be able subscribe a kafka topic

- Confluent is used as kafka provider (and its security mechanism - sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule)

- Zeppelin has to deploy jobs to cluster and NOT execute them standalone

- The kubernetes cluster doesnt use level 4 load balancer, so we cant simply use the following helm chart (charts/microsoft/spark)

- Kubernetes Level 7 Loadbalancers are used

- Basic neccessary setup (Nodes needs to have the corresponding spark versions deployed) - Acceptance criteria: I have a structured streaming script, which we can use to check if setup works, in the meantime you can use your script for development.

- Kubernetes Version is v1.16.3. - I cant grant you access to the cluster nor to the confluent stream.

Thanks for your time.
About the recuiter
Member since Mar 14, 2020
Economics Assoc
from Bihar, India

---

---

Looking for help? Checkout our video tutorial
How to search and apply for jobs

How to apply? Do you have more questions about the Job?
See frequently asked questions