Senior Systems Engineer

This is a remote position day to day with 2-3 monthly meetings at the customer’s Washington DC OR Reston, VA office. Required Skills • Datadog Administration experience on Linux platform to instrument Java based applications running on Tomcat Application Server. • Configuration experience in Infrastructure Monitoring, Network Monitoring and Centralized Logging or similar administration experience with ELK Stack – Elasticsearch (search and analytics engine), Logstash (ingest pipeline) and Kibana (visualization and creating dashboards). • Strong Linux platform (Red Hat) background. • Automation experience with scripting (Python, Shell, ANSIBLE) preferred. • Understanding of SSL setup on Linux servers. Installing CA certs etc. • Experience with Network Monitoring and knowledge on Network components like Switches, Routers, Palo Alto Network utilization SNMP, F5 Load Balancers, WebSEAL, Info Blocks, Gigamon, Network Mapping is a plus. • Working knowledge of other monitoring tools like Big Panda, CloudBeat (Synthetic Monitoring) is desired. These tools are used to monitor applications and business transactions that impact the business and customers, currently. • Responsibilities include script writing, installing, managing, and maintaining the monitoring tools, as needed, as well as integration with other tools and collaboration with other groups and their tools. TASKS • Manages, configures, and maintains the Data Dog tool on Linux platform. • Responsible for Network Monitoring, Infrastructure/Server Monitoring (Linux, Windows, AIX) using Data Dog, Application, SNMP, and Log Monitoring. • Configure centralized logging of all logs from different sources like WebSphere / Tomcat and IHS Webservers on AIX servers to Datadog on Linux. Knowledge of Load Balancers like F5 to route logs to Log server. Handling different types of Log formats. • Creates required dashboards with data visualization in Datadog. • Manages, configures, and maintains the Datadog APM tool on Linux platform. • Responsible for Java Applications’ instrumentation with Datadog, set up health rules and fine tune monitoring in Datadog. • Setup End User Monitoring / Browser Real User Monitoring of Datadog for applications, using Java script injection. • Creates Selenium scripts to monitor business transactions using CloudBeat’s Synthetic Monitoring. • Provides support to all significant production issues. Activities may include gathering information from a wide variety of sources across all platforms to analyze for correlations, identifying specific performance causes, recommending a variety of possible solutions to remedy issue and issue reports with key findings and next steps. • Creates documentation to support the management and maintenance of Datadog / Datadog tools. Provides training on tools and the associated processes and procedures. • Analyzes tool data and usage. Communicates weekly with management verbally and via written detailed status reports regarding potential problems and concerns. • Works with different Systems and Application Architecture teams to ensure that systems monitoring requirements are addressed early in the development process. Coordinates with project teams to ensure that monitoring of new applications is available before release for production. • Assists in reviewing and analyzing business & system requirements and specifications for systems monitoring tool protocols and future tool usage. Specific Required Skills • 5-8 years strong IT experience and strong working knowledge of a variety of technology platforms in a distributed environment including: Microsoft systems (e.g., Windows 2012 and 2016 Server, Active Directory, Exchange, SharePoint), Linux/Unix, VMWare, SQL Server, database architectures, TCP/IP, VPNs, Mainframe, LAN/WAN technologies and architectures. • A minimum of 3 years hands-on experience installing, integrating, managing, and maintaining monitoring tools like Datadog administration and support or similar Log Management experience with ELK Stack – ElasticSearch (search and analytics engine), Logstash (ingest pipeline), and Kibana (visualization and creating dashboards) • Experience writing Shell, Python, Selenium, VuGen scripts. • Experience with SSL certs, encryption methods on Linux • Experience developing and implementing systems monitoring and alerting strategies in diverse, large-scale environments. • Experience developing and documenting processes, procedures, and policies for tool usage and integration. • Author tool maintenance and training documentation as well as support requests for training on tool usage • Knowledge and experience with configuring alerts, dashboards, and ad-hoc reports. • Strong understanding of service level management (SLAs, SLRs, etc.) • Determine and document tool backup and recovery procedures. • Experience with data management tools and databases (e.g., DB2, SQL -familiarity desired) • Experience in systems and Java applications troubleshooting using monitoring tools like Datadog. • Understanding and experience with both waterfall and agile Software Development Life Cycles (SDLC) • Bachelor of Science in Computer Science or related field (i.e., Engineering, Applied Science, Math, etc.) or equivalent experience.