Note The job is a remote job and is open to candidates in USA. Beacon Hill is a premier National Information Technology Staffing Group, and they are seeking a Senior Data Engineer to build and support production-grade data pipelines. The role involves migrating existing data assets to Databricks Unity Catalog, optimizing pipelines, and ensuring data quality and reliability. Responsibilities Migrate existing data assets and pipelines to Databricks Unity Catalog Refactor and optimize existing pipelines using modern engineering standards and best practices Build and maintain scalable PySpark-based data pipelines Develop new source-of-truth datasets and modernize legacy data workflows Design and implement workflow orchestration using Databricks Jobs Deploy and manage pipelines using Databricks Asset Bundles Ensure data pipelines are reliable, observable, and production-ready Implement data quality controls, monitoring, and operational alerting Support platform migrations, testing, validation, and production cutovers Partner with engineering teams to improve platform stability, scalability, and data reliability Skills Strong hands-on experience building and supporting production-grade data pipelines using PySpark and Apache Spark Extensive experience working within Databricks environments Experience with Databricks Unity Catalog migrations and modernization initiatives Experience with Databricks Unity Catalog governance and security models Strong understanding of ETL/ELT design, data modeling, and incremental data processing Experience modernizing legacy data platforms and data migration projects Experience working with large-scale datasets in cloud-based data platforms Experience with workflow orchestration, scheduling, and pipeline automation Knowledge of cloud data platforms such as Snowflake and/or Google Cloud Experience implementing data quality checks, monitoring, logging, and alerting Strong troubleshooting, collaboration, and problem-solving skills Familiarity with advertising, analytics, measurement, or large-scale data ecosystems Experience with CI/CD and DataOps best practices for data engineering environments Benefits Medical, dental, vision, and federal and state leave programs as required by applicable agency regulations to those that meet eligibility Upon successfully being hired, details will be provided related to our benefit offerings Company Overview Beacon Hill offers business consulting services. It was founded in 2000, and is headquartered in Boston, Massachusetts, USA, with a workforce of 1001-5000 employees. Its website is https//bhsg.com. Apply To This Job
