Middle Data Engineer
Job ID: 200646
We are currently looking for a Middle Data Engineer (DBE). The team builds platforms to provide insights to internal and external clients of customer’s businesses in auto property damage and repair, medical claims, and telematics data. The customer’s solutions include analytical applications for claim processing, workflow productivity, financial performance, client and consumer satisfaction, and industry benchmarks.
Data engineers use big data technology to create best-in-industry analytics capability. This position is an opportunity to use Hadoop and Spark ecosystem tools and technology for micro-batch and streaming analytics. Data behaviors include ingestion, standardization, metadata management, business rule curation, data enhancement, and statistical computation against data sources that include relational, XML, JSON, streaming, REST API, and unstructured data. The role has responsibility to understand, prepare, process and analyze data to drive operational, analytical and strategic business decisions.
The Data Engineer will work closely with product owners, information engineers, data scientists, data modelers, infrastructure support and data governance positions. We look for engineers who start with 2-3 years of experience in the big data arena but who also love to learn new tools and techniques in a big data landscape that is endlessly changing.
Work at Exadel – Who We Are:
Since 1998, Exadel has been engineering its own software products and custom software for clients of all sizes. Headquartered in Walnut Creek, California, Exadel currently has 1000+ employees in development centers across America, Europe and Asia. Our people drive Exadel’s success, and they are at the core of our values, so Exadel is a people-first cultured company.
About Our Customer:
The сustomer is a leading provider of vehicle lifecycle solutions, enabling the companies that build, insure, repair, and replace vehicles to power the next generation of transportation. The company delivers advanced mobile, artificial intelligence, and connected car technologies through its platform, connecting a vibrant network of 350+ insurance companies, 24,000+ repair facilities, OEMs, hundreds of parts suppliers, and dozens of third-party data and service providers. The customer’s collective set of solutions inform decision-making, enhance productivity, and help clients deliver faster and better experiences for end consumers.
The сustomer’s company was ranked #17 in the Top 100 Digital Companies in Chicago in 2020 by Built in Chicago, an online community for digital technology entrepreneurs in Chicago, and was named one of Forbes best mid-sized companies to work for in 2019 – an important accolade and retention tool for the 2,600+ full-time company employees (alongside 350 dedicated contractors).
The сompany’s corporate headquarters is in downtown Chicago in the historic Merchandise Mart — a certified LEED (Leadership in Energy and Environmental Design) building that is also known to be a technology hub within the broader metro.
About the Project:
The customer has been working on the next generation analytics platform since 2018. The customer is currently working on Hortonworks and plans to move to Amazon EMR/Hadoop in 2021. The platform is where all the data is collected into one data lake so they can then perform next-generation analytics.
- Cross-product analytics
- Analytics for every new product the customer has. The analytics team’s products are how the customer’s company sells the products’ value to clients
- Quarterly business review meetings use data to explain how the product is helping clients in their business
- You’ll get to work with a cross-functional team
- You will learn the customer’s business
Project Tech Stack:
Technologies used are all open source Hadoop, Hive, PySpark, Airflow, Kafka to name a few
- 2+ years’ experience building, maintaining, and supporting complex data flows with structural and unstructural data
- Proficiency in Python and PySpark
- Hands-on experience with HDFS, HIVE, and SQOOP
- Ability to use SQL for data profiling and data validation
- Experience in Unix commands and scripting
Nice to have:
- Understanding of AWS ecosystem and services such as EMR and S3
- Familiarity with Apache Kafka and Apache Airflow
- Experience and understanding of Continuous Integration and Continuous Delivery (CI/CD)
- Understanding of performance tuning in distributed computing environment (such as Hadoop cluster or EMR)
- Build end to end data flows from sources to fully curated and enhanced data sets; this can include the effort to locate and analyze source data, create data flows to extract, profile, and store ingested data, define and build data cleansing and imputation, map to a common data model, transform to satisfy business rules and statistical computations, and validate data content
- Modify, maintain and support existing data pipelines to provide business continuity and fulfill product enhancement requests
- Provide technical expertise to diagnose errors from production support teams
Advantages of Working with Exadel:
- You can build your expertise with our Sales Support team, who provide assistance with existing and potential projects
- You can join any Exadel Community or create your own to communicate with like-minded colleagues
- You can participate in continuing education as a mentor or speaker
- You can take part in internal and external meetups as a speaker or listener. We support you in broadening your horizons and encourage knowledge sharing for all of our employees.
- You can learn English with the support of native speakers
- You can take part in cultural, sporting, charity, and entertainment events
- Working at Exadel means always upgrading your skills and proficiency, so we provide plenty of opportunities for professional development. If you’re looking for a challenge that will lead you to the next level of your career, you’ve found the right place.
- We work hard to ensure honest and open relations between employees and leadership, so our offices are friendly environments.
To apply for this job email your details to email@example.com
Why should you work with us?
As a successful, high-growth company, we know that our employees are critical to our success. This is why we encourage ingenuity, creativity and teamwork as important elements to the growth of our business. We believe that career growth and business growth go hand in hand.