Data Engineer
Work Location(s):
Atlanta, Georgia or Birmingham, Alabama
JOB SUMMARY
The Data Engineer role will be part of the Marketing Technology Solutions team within the Customer Technology Solutions Organization in Southern Company’s Technology Organization’s Electric Technology Solutions tower. Position responsibilities include Energy Information Exchange (EIX) data scheme versioning and management.
The Data Engineer will work with stakeholders – both business and Technology Organization to design and develop data-oriented solution architecture involving On-Prem and cloud (Microsoft Azure) technologies that align with and support the business and Technology Organization strategy and standards. This role will also work hands on to implement solutions. Additionally, this role will be responsible for management and alignment of solutions across all technology platforms working closely with developers & infrastructure teams to optimize application code, ensuring reusable services, optimizing performance & resource utilization, planning capacity, and ensuring application/data security.
JOB REQUIREMENTS
Education:
BS/BA required, preferably in Computer Science, Management Information Systems, Engineering or other technology related field.
Experience:
- Experience with Databricks.
- Experience with a major cloud platform, preferably Microsoft Azure and services including CosmosDB, Storage Application Services, Azure Data Factory.
- Experience architecting data-oriented solutions utilizing Cloud (preferably Azure) data architecture platforms and patterns.
- Experience optimizing cloud PaaS resources to adhere to best practices & reduce cost.
- Experience with Agile as well as DevOps, CI/CD methodologies.
- Experience developing complex logical and physical data models.
- Experience implementing lakehouses with medallion architecture, including data engineering pipelines and data storage layers.
- Experience with dynamic metadata driven frameworks and large-scale data ingestion patterns.
- Familiarity with streaming datasets and Databricks delta live tables.
- Experience with performance tuning techniques in a lakehouse environment, such as partitioning, aggregations, materialized views, etc.
Knowledge/Skills/Attributes:
- Databricks Platform: Proficiency in Databricks is essential. Should be comfortable working with Databricks notebooks, clusters, and jobs.
- Data Engineering Tools: Familiarity with tools like Apache Spark, Delta Lake, and MLflow is valuable. These technologies are at the core of Databricks’ offerings.
- Cloud Platforms: Data Bricks Engineers at Southern Company work with the Azure.
- Knowledge of cloud PaaS architecture & resources.
- Knowledge of structured & unstructured data – databases, data warehouses, and data lakes.
- Knowledge and understanding of software development technologies, methodology and associated tools.
- Excellent root cause analysis & problem-solving skills
- Strong technical writing skills preferred.
JOB RESPONSIBILITIES
- Data Pipeline Development: Design and develop robust data pipelines to ingest, store, and process data from various sources. These pipelines ensure efficient data flow for analytics and reporting.
- Data Modeling: Create and maintain data models that support data analysis and reporting. These models help organize and structure data for efficient querying and visualization.
- ETL Processes: Build and maintain Extract, Transform, Load (ETL) processes. These processes move data between systems, ensuring data quality and consistency.
- Data Lakes: Work on data lakes to optimize storage and retrieval for analytical purposes.
- Architect, develop, implement, integrate, and/or support cloud technology solutions in a manner that meets or exceeds business partner expectations for quality, cost, and timeliness.
- Quickly become the team’s SME (Subject Matter Expert) for data architecture & Databricks.
- Communicate effectively and timely with customers, team members, and management.
- Build effective relationships with peers, management, and business partners.
- Ensure required documentation for projects is developed.
- Adhere to Change Management policies and procedures.
- Respond/Resolve quickly & effectively to all operational problems.
- Be a student of the business, taking care to understand the challenges & goals of our company.
- Exhibit Our Values.
