Senior Infrastructure Engineer (Databricks)

<p><strong><span data-contrast="none"><span data-ccp-parastyle="heading 2">About </span><span data-ccp-parastyle="heading 2">QuantHealth</span></span></strong><span data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}">&nbsp;</span></p> <p><span data-contrast="auto">QuantHealth&nbsp;is a growing AI startup in the clinical trial space,&nbsp;leveraging&nbsp;AI, biomedical data, knowledge graphs, and real-world patient data to simulate and&nbsp;optimize&nbsp;clinical trials for pharmaceutical companies.&nbsp;</span><span data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}">&nbsp;</span></p> <p><span data-contrast="auto">Our platform helps customers simulate clinical trials, reduce development risk and cost, shorten timelines, and improve the probability of clinical trial success.</span><span data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}">&nbsp;<br><br></span></p> <p data-renderer-start-pos="33" data-local-id="3e3f5a1d0e75"><strong><span data-contrast="none"><span data-ccp-parastyle="heading 2">About the Role</span></span><span data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}">&nbsp;</span></strong></p> <p data-renderer-start-pos="33" data-local-id="3e3f5a1d0e75">Quant Health’s clinical trial simulations platform is composed of hundreds of Spark ETL jobs, several apps for use by internal domain experts, many streaming model inference endpoints and terabytes of patient, drug compound and clinical trial data. All of this is powered by Databricks and hosted across multiple environments in both multi-tenant and single-tenant deployment models. As Quant Health’s first infrastructure engineer focused on our internal machine learning platform, you’ll partner with Data Engineers and Data Scientists to manage our Databricks infrastructure with <span data-highlighted="true" data-vc="highlighted-text"><span data-testid="definition-highlighter">IaC</span></span>, build robust CI/CD pipelines, define our Databricks <span data-highlighted="true" data-vc="highlighted-text"><span data-testid="definition-highlighter">IAM</span></span> model and be the company’s foremost technical champion of Databricks best practices.<br><br></p> <p><strong><span data-contrast="none"><span data-ccp-parastyle="heading 2">Responsibilities</span></span><span data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}">&nbsp;</span></strong></p> <ul data-local-id="030da2f929e3" data-indent-level="1"> <li> <p data-renderer-start-pos="1034" data-local-id="a9118727ebea">Work closely with Data Engineering, Platform Engineering and Data Science teams to understand their work and the associated infrastructure requirements</p> </li> <li> <p data-renderer-start-pos="1189" data-local-id="6828e25db704">Architect, implement and own flexible <span data-highlighted="true" data-vc="highlighted-text"><span data-testid="definition-highlighter">CICD</span></span> tooling that teams developing on Databricks will use to easily deploy their workloads</p> </li> <li> <p data-renderer-start-pos="1321" data-local-id="aafefd81efd2">Design and enforce a robust data-isolation strategy on Unity Catalog</p> </li> <li> <p data-renderer-start-pos="1393" data-local-id="fe8dc56dee48">Design a company-wide <span data-highlighted="true" data-vc="highlighted-text">IAM</span> model to facilitate secure access for both human users and service principals to compute and data across all of Quant Health’s Databricks environments</p> </li> <li> <p data-renderer-start-pos="1573" data-local-id="af203561718b">Manage and monitor Databricks costs, including cluster optimization, resource tagging strategy, cost-based alerting, etc.</p> </li> <li> <p data-renderer-start-pos="1698" data-local-id="345e66358cab">Implement observability functionality for our Data Science and Data Engineering workloads, including cross-component tracing, incident management, alerting and reliability metrics</p> </li> <li> <p data-renderer-start-pos="1881" data-local-id="18ad4de447de">Create a disaster-recovery plan including backup automation, fail-over procedures and restore drills</p> </li> <li> <p data-renderer-start-pos="1985" data-local-id="a42f0fefaa8e">Drive adoption of best practices and new features on the Databricks platform<br><br></p> </li> </ul> <p><strong><span data-contrast="none"><span data-ccp-parastyle="heading 2">Qualifications</span></span><span data-ccp-props="{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}">&nbsp;</span></strong></p> <ul data-local-id="25bcd1d51262" data-indent-level="1"> <li> <p data-renderer-start-pos="2082" data-local-id="e3bd77190959">At least 6 years of experience in each of the following: DevOps, managing complex cloud platforms (Databricks, Snowflake, AWS, GCP, Azure, etc.), <span data-highlighted="true" data-vc="highlighted-text">IaC</span> tools (Databricks Asset Bundles, Terraform, Cloud Formation, etc.), Relational DB Management, multi-tenant and single-tenant deployment models</p> </li> <li> <p data-renderer-start-pos="2378" data-local-id="9c0e09264be1">At least 2 years of experience in each of the following: direct collaboration with Data Scientists or Machine Learning Engineers, <span data-highlighted="true" data-vc="highlighted-text">IAM</span>/<span data-highlighted="true" data-vc="highlighted-text"><span data-testid="definition-highlighter">RBAC</span></span>, SRE and observability platforms (NewRelic, Data Dog, etc.), Python</p> </li> <li> <p data-renderer-start-pos="2589" data-local-id="cb172fe9c194">Excellent written and verbal communication skills</p> </li> <li> <p data-renderer-start-pos="2642" data-local-id="3e923ba64527">Ability to work independently and as part of a team<br><br></p> </li> </ul> <p><strong>Advantages</strong></p> <ul class="ak-ul" data-local-id="7e0fa14c1062" data-indent-level="1"> <li> <p data-renderer-start-pos="2711" data-local-id="ed26ce54b957">Experience with Databricks: DABs, Unity Catalog, access control, metastores, jobs, apps, serving endpoints, clusters, compute policies, the Databricks Terraform provider, etc.</p> </li> <li> <p data-renderer-start-pos="2890" data-local-id="02198a5d5e77">Databricks certifications</p> </li> <li> <p data-renderer-start-pos="2919" data-local-id="176aabc371df">Prior experience in the life sciences industry</p> </li> <li> <p data-renderer-start-pos="2969" data-local-id="b401fb739844">Experience developing single-tenant solutions for large enterprise clients</p> </li> <li> <p data-renderer-start-pos="3047" data-local-id="70143a83326f">Working on or closely alongside Data Engineering teams and with data warehouses, data lakes, etc.</p> </li> <li> <p data-renderer-start-pos="3148" data-local-id="e8732ccd626e">Working knowledge of PySpark, Spark, distributed computing</p> </li> </ul> <p>&nbsp;</p>