With a mission to help countries in the Asia-Pacific region fight poverty and build sustainable infrastructure, the Asian Development Bank relies as much on technology as on its thousands of employees.
Since its inception in 1966, ADB has consistently pushed the boundaries of technology use by its workforce. This led to a decision in 2016 to improve mobility and cloud-based collaboration – an effort that would be appreciated when the COVID-19 pandemic hit in 2020, forcing a shift to remote working.
ADB’s initial IT modernization project involved a transition from Lotus Notes to Office 365 for collaboration, replacing desktop computers with laptops, and moving from BlackBerry devices to iOS devices. The organization also replaced its secrets engine, Lotus Notes ID Vault, with the open source version of HashiCorp Vault to store passwords, API keys, tokens and certificates.
After completing the project, ADB executives realized that employees continued to struggle with collaboration, data management, and general efficiency. A second, larger IT modernization initiative was launched a few years later. “We generated about five Eiffel Towers per year in paper consumption,” said Krista Camille Lozado, ADB’s senior IT specialist for innovation and engineering. “We are a development bank, but we couldn’t keep pace.”
Then the pandemic hit, testing ADB’s capabilities. While the organization has adapted to many challenges, employees have still encountered many problems. Patchy connectivity, network slowness and latency, security and a complex disaster recovery situation complicated everything.
Fortunately, ADB reacted quickly. The bank tripled its IT modernization investment from $6 billion to $20 billion virtually overnight.
ADB had ambitious goals for its new IT modernization project:
- accelerate migration to the cloud while increasing security, speed and resiliency;
- build new infrastructure using the bank’s existing framework;
- running data centers in multiple locations around the world; and
- improve disaster recovery.
The IT modernization project involved automation improvements.
After deciding to standardize on Microsoft Azure, ADB’s first step in advancing its automation capabilities was choosing the right tools. The team opted for three systems: the agentless Red Hat Ansible IT automation engine, HashiCorp Packer for creating virtual machine images, and HashiCorp Terraform for infrastructure development and manipulation.
All three tools have their roots in infrastructure as code, a method that manages and provisions infrastructure through code through a graphical user interface instead of resorting to more manual processes. Adoption of these types of tools was important because they promote collaboration and reuse, Lozado said.
“We wanted to write our code so that everything was modular,” she explained. “For example, in Terraform there would be a module representing an infrastructure or resource item, so there is a declarative module to say, ‘This is a SQL server in Azure’ or ‘This is a Windows virtual machine in VMWare”. ‘ Modules are like Lego blocks.
At the same time, the team opted to move from open source versions of its HashiCorp tools to enterprise versions, which offered broader functionality and support. Terraform Enterprise, for example, allows developers to ensure that all Terraform runtimes use ADB modules. The modules already have all the security protocols built in, Lozado noted.
Terraform is also an important infrastructure building tool for ADB. Everything the organization needs to get started is done in Terraform. The team uses Ansible for configuration management and other IT automation tasks such as application deployment and orchestration. Lozado put it this way: “If ADB were building a restaurant, the construction workers would be Terraform and the people running the restaurant would be Ansible.”
As the project came together, the team began to see the benefits of how the automation built on itself. For example, the IT team created a playbook detailing maintaining and powering servers, but when the security team embraced security orchestration, automation, and response (ASCEND) to identify zero-day vulnerabilities, the IT team had a brainstorm. “Since we already had a mechanism to fix servers and the SOAR team now wants us to fix servers when they identify something, why not have SOAR fix it automatically by calling an Ansible API?” Lozado said. “We’ve already done the work, so it was an easy call.”
With automation efforts under control, it was then time to pursue a serious migration of ADB’s data centers to the cloud. While ADB originally planned to set up a new data center using Azure in the Singapore region closer to its headquarters in the Philippines, the spread of COVID-19 demanded a different strategy. The new data center is virtual, Azure-based, and built by ADB using Terraform, Ansible, and Microsoft Azure DevOps Server. Although complex with over 2,000 assets, the virtual data center was up and running in less than five days. The ADB team had all the assets configured in less than 45 days.
In addition, the data center allows users run a new VM in less than two minutes.
Disaster recovery moves to the cloud
AfDB has used complicated disaster recovery processes that produce unsatisfactory results. For example, the team only runs DR procedures or activations twice a year, but it’s never seamless because dependencies or updates haven’t been done properly. “There were always pieces missing because we forgot something,” Lozado said. “But the biggest problem we had is that because the knowledge is siloed, one team might have uncovered something, but the other teams would have no way of knowing.”
The team decided to standardize, automate and centralize their disaster recovery processes, all without compromising security or resilience. The project began with the selection of a secondary site. It was timely, since the AfDB headquarters based in Manila is overdue for an earthquake. As a temporary measure before moving DR to Azure, the team chose a site in Valencia, Spain. And instead of relying on a vendor-developed DR offering, the ADB team opted to develop their own, relying on techniques such as data replication. Eventually, all disaster recovery will be done in the cloud, using native, immutable cloud infrastructure.
In addition to moving disaster recovery to the cloud, ADB’s IT modernization project aims to improve data management, governance and storage. “We know we have the data, but we keep repeating the same mistakes over and over because a lot of the knowledge stays in someone’s inbox and we never find it,” Lozado said. “There is no concept of a single version of the truth.”
About the AuthorKaren D. Schwartz is a technology and business writer with more than 20 years of experience. She has written on a wide range of technology topics for publications including CIO, InformationWeek, GCN, FCW, FedTech, BizTech, eWeek, and Government Executive.