Data Engineer
Job Req 1. Programming language a. Phyton b. Advance SQL (subquery, ROW_NUMBER, RANK, PARTITION BY, Indexing, Explain Plan, dan Query Tuning, Stored Procedures, Triggers, View, Materialize View, Constraints, indexing, query optimization) 2. Database Systems a. MySQL b. PostgreSQL c. Oracle d. Snowflake e. Unstructured Data 3. Data modeling dan arsitektur a. Distributed Data Storage Systems (partitioned data storage, sorting key, SerDes, data replication, caching and persistence) b. Distributed Data Processing Systems (partitioning, predicate pushdown, sort by partition, maintaining size of shuffle blocks, window function, leveraging all cores and memory available in the cluster to improve concurrency) c. Stream Data Processing (Real-time, Stream and Batch Processing) 4. Tools a. Data Pipelines and Automation (Airbyte) b. Data ingestion in Message Queues c. Data Wrangling operations – Pandas,numpy,re d. Data Scraping – requests/BeautifulSoup/lxml/Scrapy e. Interacting with External APIs and other Data Sources, Logging f. Parallel processing Libraries – Dask, Multiprocessing g. Data engineering tools (Apache Kafka, Apache Airflow) h. Cloud Native Technologies (Serverless Computing, Virtual Instances, Containering – Docker, Orchestration -Kubernetes) i. Data Visualization (Qlikview, Tableau) Job Desk 1. Membuat perencanaan aktivitas pekerjaan. 2. Membuat stream pipeline for realtime data processing. 3. Membuat datamart untuk report dan dashboard. 4. Mengumpulkan, mengurutkan, membersihkan, dan mengelola data. 5. Melakukan analisis dan memberikan hasil analisis kepada tim. 6. Mengelola data transfer (tally data) dari database transaksi ke data warehouse. 7. Mengelola server job scheduler. 8. Berkolaborasi dengan tim untuk mengidentifikasikan dan memperbaiki masalah teknis pada backend dan frontend. 9. Membuat laporan pekerjaan bulanan.
