Hi, what I need is the following:
- I need a helm script (kubernetes) which 'installs' the latest spark version as yarn cluster on kubernetes
- This setup should provide Zeppelin
- All of this has to work with sparks latest structured streaming framework and should be able subscribe a kafka topic
- Confluent is used as kafka provider (and it's security mechanism - sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule)
- Zeppelin has to deploy jobs to cluster and NOT execute them standalone
- The kubernetes cluster doesn't use level 4 load balancer, so we can't simply use the following helm chart (charts/microsoft/spark)
- Kubernetes Level 7 Loadbalancers are used
- Basic neccessary setup (Nodes needs to have the corresponding spark versions deployed) - Acceptance criteria: I have a structured streaming script, which we can use to check if setup works, in the meantime you can use your script for development.
- Kubernetes Version is v1.16.3. - I can't grant you access to the cluster nor to the confluent stream.
Thanks for your time.
About the recuiterMember since Mar 14, 2020 Monika Putri
from Narino, Colombia