Workload Scheduler Administrator / Infrastructure Engineer

IBM Workload Scheduler AdministratorWe are seeking a highly skilled IBM Workload Scheduler (IWS) Administrator to manage, maintain, and optimize our partner's enterprise batch scheduling infrastructure. The successful candidate will be responsible for the end-to-end administration of the IWS environment hosted primarily on Red Hat Enterprise Linux (RHEL). This role requires a strong blend of IWS expertise, Linux system administration, and scripting to ensure high availability and seamless execution of critical business workloads.ResponsibilitiesAdminister Production IBM Workload Scheduler (aka Tivoli Workload Scheduler) environment with 28,000 unique daily jobs across about 350,000 daily job runs, 44 servers, and three other change control environmentsAdminister, install, configure, and patch/upgrade IWS components (Master Domain Manager, Dynamic Agents, Dynamic Pool, Dynamic Workload Console)Work with Product Owner on communicating work streams in JiraManage job promotions with Workload Application Template-based process, ensuring safety checks for platform stability assessed on each job promotionManage change control of four (4) separate change control environments, enforce change control standards and policiesMaintain and continuously promote 99.17% Production platform uptime per calendar month (uptime calculated excluding planned change control outages, planned weekly maintenance windows) using SOP's, DevOps tools, and disciplined change control across change control environmentsOrderly communication of platform-impacting news to user community of about 500 developers and data engineersProduction consists of 44 servers between MDM, DWC, and agentsResolve complex job failures, performance bottlenecks, agent issues, and infrastructure issuesAdvise on complex job scheduling design questions that less experienced scheduling support teamMonitor the health of the scheduler environment, manage database maintenance, and perform backup/disaster recovery, and monthly failoversDefine and maintain security policies, user authorizations, and authentication for the DWCRespond to Cybersecurity vulnerability assessments and PCI (and other regulatory) audit inquiriesDesign and implement Ansible automation and self-healing mechanisms to continuously reduce unplanned outagesCoordinate with offshore performing SOP's during non-working hoursScripting Python with IWS REST APIRequirementsUS CitizenshipStrong experience with IBM Workload Scheduler architecture, especially Dynamic Workload Broker, V10.1+, high availability of MDM's managing Fault Tolerant Agent and Dynamic Agent agent architecturesStrong conceptual understanding of Master Domain Manager (MDM), Backup MDM (BMDM), Dynamic Workload Console (DWC), Fault Tolerant Agent (FTA), Dynamic Agent (DA)Strong grasp of conman CLI to monitor and control production plan, check job/job stream/resource statusStrong grasp of composer CLI to define, modify and extract scheduling objectsStrong grasp of planman CLI to control pre-production plan and GUI mirroringStrong grasp of lifecycle of daily production planning process, phases of JNextplan/FINALProficiency in navigating the DWC web-based GUI to monitor workloads, manage user access security, and define scheduling objectsExperience installing IWS components, applying Fix Packs, and Interim FixesTroubleshooting with logs under TWSDATA/stdlist, adjusting trace level for netman, batchman, writer, mailman, etc.Strong experience with IBM WebSphere LibertyStrong grasp of reading messages.log, traces.log, FFDC logsStrong grasp of configuring JVM heap sizesStrong grasp of configuring tracing scope, tracing levels, tracing retentionStrong experience with Red Hat Enterprise Linux 8+Deep familiarity with bash/shell commands for text processing (for example, grep, awk, sed), file manipulation, and system navigationAbility to manage, start, stop, and troubleshoot SystemD services using systemctl and journalctl for IWS agents and MDMManaging user accounts, groups, service accounts and deep knowledge of Linux file permissions (chmod, chown, ACL on local filesystems and NFS)Ability to monitor system performance using tools like top, htop, vmstat, iostat, and sar to troubleshoot bottlenecks and platform unresponsivenessUnderstanding of Logical Volume Manager (LVM) and filesystem usageChecking TCP port availability, firewall rules (firewalld/iptables), and connectivity between MDM and Dynamic Agents using netstat, ss, ping, curl, etc.Managing SSL/TLS certificates, private keystores, public truststores, and working with Certificate AuthorityStrong experience with scripting (Bash Shell, Python, etc.) for automationUnderstanding of networking principlesUnderstanding of basic Oracle database administration, enough to troubleshoot with DBA's to prove when an issue is in OracleUnderstanding of basic SQL to query job metadataUnderstanding of checking database connectivityUnderstanding of AWS cloud infrastructureExperience with using secrets manager (CyberArk PPM, Hashicorp Vault, or similar)About UsFor more than 20 years, NewGen Technologies has solved our clients' toughest IT challenges with integrity, security, and outstanding service by delivering both technology and talent.

We have helped secure borders, have used artificial intelligence (AI) to fight terror, aided the identification of criminals, and have helped to prevent crime through the introduction of biometrics. Our team of Highly Cleared Specialists have hard-to-find skills and expertise in a wide spectrum of technologies to provide solutions that transform business processes and solve problems of national significance.