• Serve as Splunk engineer, senior leader and/or subject matter expert (SME) responsible for planning, designing, and implementing Splunk across multiple enterprise networks cluster implementations
• Assesses current Splunk implementations for each network and recommend changes to distributed deployments to include Indexer Clustering, Search Head Clustering, Forwarders, daily indexing, search volume, number of data sources, number of users, custom apps/dashboards/visualizations
• Monitor, troubleshoot, and analyze overall health of Splunk infrastructure
• Perform root cause analysis, recommend, and implement tactical and strategic solutions to problems
• Develop, update and document Splunk architecture, operational processes, and training materials
• Ability to automate global, multi-site solutions with Ansible, Python, and Bash scripting techniques
• Experience with various log ingestion methods, new data onboarding and related products, such as Log Agents, syslog, DB Connect (dbConnect), Universal Forwarder (UF) Agent, HTTP Event Collector
• Working knowledge of Linux; general networking topics such as SSL, load balancing, routing protocols, firewall rules, and ability to support/interact with McAfee Endpoint Security System (ESS) for RHEL
• Document steps required to design/engineer Splunk systems for each network to include virtual/real IP address, Fully Qualified Domain Name (FQDN), DNS entries, Role Based Access Controls (RBAC), service accounts, web certificates, licenses and physical/virtual location of each component
• Candidate will oversee activities to include planning, researching, deploying, monitoring, upgrading, patching, and troubleshooting Splunk components spanning a large and complex environment
• Ability to maintain valid system certificates, application certificates, F5 load balancing local traffic management (LTM) and two-factor authentication (2FA) within a smart card environment
• Ability to take bootstrap ideas to polished, efficient dashboard
• Sr Systems Engineer/SME/Architect/Developer provides tech support in system architecture, system design, system integration & technical management
• Review existing data models with special attention to the following data models, Identity Management Authentication, Malware, Endpoint, Network, Traffic, Risk, Threat Intelligence, among other data models/deprecated models.
• Provide best practice recommendations: how to update/maintain/add new Data Models; Data Model Creation/Acceleration/Maintenance; Risk Based Alerting; Scaling of Correlation Searches
• Oversee the baseline configuration, fine tuning data models, ensure operational data integrity, and using vendor best practices for the Splunk systems and secure management across multiple unclassified and classified network locations supporting the interaction with Tenable products within Assured Compliance Assessment Solution (ACAS) including .SC (SecurityCenter™) and Nessus® Scanner™
• Ability or experience in evaluating scan report data from Tenable Nessus; participate in the review and response phases of the Vulnerability Management (VM) life cycle
• Install and patch operating systems, applications, and document Department Information Systems Agency (DISA) Security Technical Implementation Guidelines (STIGs) checklists applicable to each Non-classified or Secret Internet Protocol (IP) Router Network (NIPRNet, SIPRNet) network environment for all Splunk implementations
• Assist in the Splunk system installation/maintenance of configuration files, custom security policies
• Manage or assist the processes related to onboarding users/projects, configuration audits, building data models, summary data reports, basic Search Processing Language (SPL), advanced search analytics
• Ability to create Splunk network designs diagrams with Microsoft Visio (include specialty requirements)
• Implement/create report dashboard designs, automated custom email report notifications, report log data repositories for each environment that are specific to the following audiences: Leadership & Executives; Cybersecurity Staff; and System Administrators
• Ensures networks receive periodic updates from AFCYBER-released software patches, updates, and upgrades via Time Compliance Technical Orders (TCTO), Time Compliance Network Orders (TCNO), Maintenance Tasking Order (MTO) and Notices to Airman (NOTAMs)
• Assist AF Cyber personnel with the DISA Information Assurance Vulnerability Management (IAVM) programs, cybersecurity toolsets, and Operation Order (OPORD)/Fragmentary Order (FRAGO) support
• Ensures external networks receive inventory data for compliance data DoD Enterprise Logging Ingest, NiFi, and Cyber Situational Awareness Refinery (ELICSAR) Big Data Platform (BDP)
• Communicate, manage expectations, eliminate gaps and successfully interact with multiple external and internal 26^th NOS team leads, administrators, analysts, users, customers, system owners and management
• Guide customers in the use of strategic products through education and guidance, first-use and tuning assistance problem solving and critical situation resolution.
• Candidate will be a part of the 26^th NOS Enterprise Networking Application Tools (ENAT) team which will be small but highly visible so experience in at least one of the other monitoring platforms or enterprise tools is helpful if not critical (SolarWinds Orion, CA’s NetQoS NetFlow Analysis, Cacti, F5 Big-IP Appliance)
• Candidate will report to the 26^th NOS Systems Administration (SA) team leadership

Qualifications

• SrSA/Engineer/SME/Architect/Developer candidate must have a minimum of 6+ years of Splunk products experience and/or enterprise monitoring tools experience interacting with 3^rd party systems preferably in role(s) such as a system administrator, engineer, developer or architect capacity
• Splunk experience with design, implementation and administration in a large-scale environment preferably overseeing daily, weekly, monthly functions and best practices
• Identify, analyze, define, & coordinate user, client, and stakeholder needs and translate them into technical requirements
• Support day-to-day technical communication systems and incident tickets in support of operations
• Candidate should have 4+ years of years of hands-on experience in:
  • System Integrator and/or administrator for Splunk users, searches/reports, dashboards, systems or 3^rd party onboarding log data
  • Windows OS, UNIX or Linux-based systems support with experience in mid-to-large data center environments and patch/update management
  • Demonstrated advanced diagnostics, analytical, troubleshooting skills
• Preferred system hardening experience
• Strongly preferred Splunk Enterprise Security experience
• Perform systems analysis, design review, integration of complex system applications
• Experience with disaster recovery (DR) - expertise in risk reduction, hot/warm site DR architecture
• Experience with physical servers and within virtualized environments such as VMware vSphere’s vCenter Server Appliance, ESXi hosts, virtual machines (VMs), SAN datastores, host bus adapters (HBA) fiber connectivity, and/or VM/Host distributed resource schedules (DRS) groups/rules
• Scripting experience with regular expressions and languages such as: Ansible, Bash, JavaScript, HTML, Perl, PowerShell, or Python
• Knowledge of data communications, local-area networking (LAN), wide-area networking (WAN), servers, routers, switches, and firewalls
  • Network (Layer 2, 3) LAN/WAN knowledge and switches/routers
  • Thorough understanding of Internet Protocol (IP) routing, switching, and OSI model

 

Clearance Level
Secret
Certifications
CompTIA Security+ ce (continuing education) or (ISC)² CISSP One Operating System Certification: Comp