Spark-Scala Lead Developer

You will be part of a product development team, reporting to the SVP of Engineering, with a goal of finding innovative solutions to processing and reading vast amounts of raw data from various systems and various formats using spark. This involves advanced data pipelines that will be embedded into our product.

 

Responsibilities:

– Design & develop Scala/Spark processes for data discovery

– Produce unit tests for Spark transformations and helper methods

– Write Scaladoc-style documentation with all code

– Design data processing pipelines

 

Skills Required:

– Scala (with a focus on the functional programming paradigm)

– Apache Spark 2.x

– Apache Spark RDD API

– Apache Spark SQL DataFrame API

– Apache Spark Streaming API

– Containerization experience (docker & Kubernetes)

– Spark query tuning and performance optimization

– SQL database integration (Microsoft, Oracle, Postgres, and/or MySQL, etc.)

– Experience working with HDFS, S3, Cassandra, and/or DynamoDB

– Experience with document processing under Spark Streaming

– Experience with Kafka & Zookeeper

– Understanding of distributed systems

Job Category: Development
Job Type: Full Time
Job Location: Costa Rica Remote

Apply for this position

Allowed Type(s): .pdf, .doc, .docx
Would you like to book a free consultation with an expert?Do it now!
>