We are looking for a Medior/Senior Big Data Software Engineer for our client in Rotterdam. If you’re looking for an opportunity with a scale-up growing company, have knowledge of and experience with Big Data, Apache Spark and Machine Learning, then we are looking for you!
Our client develops unique tools to make markets transparent fast. They gather and maintain data from diverse public places, which is valuable information to their clients in order to achieve their business goals. With their own tool and SaaS marketing platform, they are unique in the field of sales and marketing.
Job description Big Data Software Engineer
As a Big Data Software Engineer, you will be part of the “data” team that is responsible for continuously developing and deploying tools for the big data pipeline. The technology stack includes Spark, MLlib, Java, Python, Airflow, Hadoop, Parquet and different database solutions (SQL and NoSQL) running in the cloud. They use a microservices and containerization (Docker, Kubernetes) approach to develop new solutions.
You will develop, maintain and improve the big data pipeline within the scope of:
- data ingestion (crawlers, external API’s, data dumps)
- stream/batch processing the newly ingested data (Kafka, Camel)
- storing the raw data (Parquet, Hadoop)
- cleaning/parsing/matching/combining the data (Spark, MLlib, SQL)
- storing the cleaned data (Elasticsearch, Cassandra, Neo4j)
- disclosing the data via secure API’s ready to use by our internal products and external partners.
- security, authentication and authorisation
- distributed systems, microservices, databases and NoSQL technologies
Skill set Big Data Software Engineer
- You have a bachelor’s or master’s degree (e.g. Computer Science, Software Engineering)
- You have good knowledge of Java, Scala and/or Python
- You have experience with Apache Spark
- You have good SQL & NoSQL skills
- You have good communication abilities and capacity for teamwork
- You have excellent skills in writing and speaking in English
- Experience with Scrum
Preferred
- Experience with developing Big Data ETL pipelines
- Experience with Cloud Platforms, like GCP, AWS or Azure
- Experience with CI/CD tools
- Foundation of Machine Learning and basic algorithms.
Compensation Big Data Software Engineer
- You will work in an energetic and inspiring working environment. A brand new office in the city centre of Rotterdam.
- Good salary, depending on your experience
- Office presence and homework can be combined
- Small entrepreneurial team
- Lots of technical freedom