Job Description
- Understand the design specifications.
- Follow the best practices for core product development.
- Data ingestion for distributed system with parallel processing, in Java, CPP or Golang for
- from cloud storage – Amazon S3/Azure Cloud Storage/Google Cloud Storage, relational database – Snowflake, Google BigQuery, PostgreSQL, Files, Kafka, data lakehouse (Iceberg)
- High-availability (HA) loading, cross-region replication
- Loading monitoring/error reporting
- Spark Connector
- Third-party management – Kafka, Kafka Connect
Requirements
- building distributed systems applying applying parallel processing.
- Any or of the following – Kafka, Zookeeper, Spark, Stream processing.
- Experience in Kafka Connect, Kafka stream, Kafka security and customization
- Spark connectors
- Event driven architectures.
- Agile development
- CICD
Nice to have:· gRPC protocol experience and multi-threading· Zookeeper/ ETCD/ Consul· Familiarity with distributed consensus algorithms (Paxos / Raft)· Docker, Kubernetes
Job Type
Payroll
Categories
Back End Developers (Software and Web Development)
Principal Engineer (Software and Web Development)
Data Scientist (Software and Web Development)
Software Engineer (Software and Web Development)
Data Engineer (Software and Web Development)
Product Development Engineers (Engineering)
Must have Skills
- CI/CD – 4 YearsAdvanced
- Agile Methodology – 3 YearsIntermediate
- Distributed Computing – 2 Years
- Google Cloud – 4 YearsIntermediate
- Azure – 4 YearsAdvanced
- AWS S3 – 4 YearsAdvanced
- BigQuery – 3 YearsIntermediate
- Snowflake – 3 YearsAdvanced
- Apache Spark – 3 YearsIntermediate
- Apache-Kafka – 4 YearsAdvanced
- Go Lang – 3 YearsAdvanced
- PostgreSQL – 4 YearsAdvanced
- C++ – 3 YearsAdvanced
- Java (All Versions) – 4 YearsAdvanced
- CPPUnit Testing