Splunk Admin

For more than half a century, UCLA Health has provided the best in healthcare and the latest in medical technology to the people of Los Angeles and throughout the world. Comprised of Ronald Reagan UCLA Medical Center, UCLA Medical Center, Santa Monica, Resnick Neuropsychiatric Hospital at UCLA, Mattel Children's Hospital UCLA, and the UCLA Medical Group with its wide-reaching system of primary-care and specialty-care offices throughout the region.

* Responsible for systems performance & capacity planning for all CareConnect and Enterprise specific infrastructure components * Provide hands on direction as well as thought leadership to ISS teams * Candidate must be able to clearly illustrate and explain standards of excellence for developing best practices in areas of Capacity & Performance. * Standardize Splunk forwarder deployments, configurations and maintenance across a variety of UNIX, Windows and Virtualization platforms * Troubleshoot Splunk server and forwarder problems and issues * Designing and maintaining production-quality dashboards and reports customization * Assist internal users of Splunk in designing and maintaining production-quality dashboards * Provide Splunk training to internal customers as required * Participate as an escalation point for 24x7 operations support for Splunk * Monitor the Splunk infrastructure for performance and capacity planning and optimization * Support Splunk on Linux, Windows and virtualization platforms * Proven track record of automating processes and procedures * Splunk experience to support Technical and Development teams in monitoring server/application performance, capturing metrics and developing/presenting reports. * Provides input and guidance to the build teams, ISS Infrastructure Operations, and the CareConnect technical and Application Architects * Define and implement Capacity and Performance Management methodology, processes and procedures including: develop baseline model, lead capture and validation of business Volumetric, implement process for ongoing business Volumetric capture * Design, support and maintain the Splunk infrastructure in a highly available configuration * Implement feedback loop for calibration of baseline model with actual measurement captured through performance testing * Participate in infrastructure and applications design reviews * Select, install and configure performance and load testing measurement tools in conjunction with IT Operations * Design performance and load testing environments based on non-functional requirements and business metrics * Coordinate internal ISS CareConnect performance testing and metrics with Epic performance testing team * Plan and execute end-to-end and system component performance and scalability tests across all relevant environments and systems * Analyze, document and publish performance test results * Conduct root cause analysis for performance and scalability problems * Lead resolution of performance and scalability problems * Provide input into all major application solution decisions * Coordinate with the Build and Architecture Teams to provide guidance in the configuration of the Epic solution * Develop and manage cross-training plan.

* Bachelor's degree (ideally in IT field) or at least 5 year's experience in Capacity & Performance management. * Minimum of 5 years performance measurement and management experience with integrated enterprise packaged software solutions. * Minimum of 5 years experience in System architectures: * Knowledge of storage subsystems, * IO subsystems, * Performance and capacity modeling, * Tuning and reporting. * Advanced knowledge of Epic application and Cache database architecture and deployment, Chronicles and routines is highly desired. * Experience with Oracle and SQL Server. * Experience in integration design supporting mission-critical, 24x7 business processes. * Experience with system capacity monitoring and forecasting. * Advanced level experience in performance scripting. * Advanced level experience in Splunk reporting. * Must have successful track record of developing effective QA measures that include capacity management and reporting. * Knowledge of instrumenting virtualization platforms (UCS/Hyper-V platform desired) using Splunk. * Experience with defining and documenting Capacity & Performance Management methodology, processes and procedures. * Experience with managing the development of performance and load test scripts including creation of the test plan and test scenarios whose scope includes new code or changes in existing code and new technologies using LoadRunner. * Understanding of information systems in the healthcare operations domain is preferred. * Proficiency in working with leading edge, automation and design using the following: * storage capacity/performance tools: * Splunk * Design, support and maintain the Splunk infrastructure in a highly available configuration * Standardize Splunk forwarder deployments, configurations and maintenance across a variety of UNIX and Windows platforms * Troubleshoot Splunk server and forwarder problems and issues * Designing and maintaining production-quality dashboards * Assist internal users of Splunk in designing and maintaining production-quality dashboards * Participate as an escalation point for 24x7 operations support for Splunk * Monitor the Splunk infrastructure for capacity planning and optimization * Design core scripts to automate Splunk maintenance and alerting tasks * Support Splunk on Linux and Windows-based platforms * Proven track record of automating processes and procedures * Nagios * System Pulse * LoadRunner v12.0 (preferably Citrix protocol) * Storage stress testing * Perform SQL Server data interpretation * EMC ECC Perf Manager * EdgeSight **Critical Success Factors:** * Excellent conflict-resolution skills. * Excellent logic, verbal and math skills; * Ability to explore and analyze alternatives. * Demonstrated ability to develop creative solutions to complex problems. * Experience in developing decision support tools to help the performance of a team. * Effective at facilitating and leading group discussions/meetings, building consensus and managing issues and risks. * Excellent Communication Skills, Strong Data Analysis/Correlation, reporting. * Strong problem solving skills. * Expert understanding of software development and infrastructure project lifecycles. * Ability to effectively multi-task across many assignments, prioritize work, and delegate tasks. * Ability to communicate with people effectively, organize resources within the team and motivate team members.