The Solution: Hitachi Virtual Storage Platform
The bank sent out requests for proposals to leading mass storage integrators, with very high performance requirements and numerous extra conditions. Requirements included the provider’s consent to make the offered systems available in advance to enable the bank to verify the parameters declared in the bid and to run comprehensive tests, primarily performance-related ones. The delivery of the systems did not mean winning the contract. It was decided that if the solution selected by means of comparing the bids failed to achieve the declared parameters, the hardware would be returned to the vendor with no extra costs for the bank. In such case, the hardware of the next provider on the list was to be tested.
The bank was perfectly aware which storage parameters were critical. The detailed requirements were prepared by a team of the bank’s experts responsible for applications, databases, networks, servers and mass storage.
“After a few months of tests of applications, system environments and mass storage, which preceded the publication of the request for proposals, we prepared a very detailed load profile for the whole installation. Quite a big team worked on it, about 10 people in total. Only after those examinations, during which we were sometimes analyzing single transactions, could we thoroughly define the actual functional and performance requirements for the new mass storage environment,” reports Urszula Michnowska-Czyż, leader of the Mass Storage Team in the IT Infrastructure Department of PKO Bank Polski.
Within those requirements, performance was key, as the bank wanted every assurance that the mass storage environment would add not more than 2.5 milliseconds to the total transaction processing time. Transaction processing delays exceeding 3 milliseconds were identified among the major causes of the slow-downs of the entire environment. As an additional condition, the bank wanted this delay threshold kept even with synchronous replication to the backup data center enabled.
“The winner of the initial stage of the procedure was Hitachi, offering VSP (Virtual Storage Platform) systems,” reports Księżycki. For test purposes, 2 VSPs were installed in the main location, with 90TB of raw disk capacity each. The resources on each storage system were partitioned to form 2 logical domains, 45TB each, in order to make local copies. An identical installation of 2 more storage systems was started up at the backup center. The distance between the locations was only 20 km, but for the tests they were connected with an optical fibre, 50 km long. “By this, we wanted to verify empirically if keeping the delay below the desired threshold was still possible with the backup site located further away than the one we use currently,” Księżycki says.
The testers installed on each VSP: 2 pairs of controllers with IBM FICON® ports for mainframe servers, 2 pairs of caching controllers (total cache capacity of 160GB), 2 pairs of disk controllers, 2 pairs of storage system managing controllers, and a pair of controllers with Fibre Channel ports for remote replication. To test the storage systems in situations as close as possible to their prospective live environment, the bank’s engineers “recorded” about 3 hours of extreme-load work of its production environment. To this end, they used the Hiperstation application by Compuware, which had been well proven for similar purposes before. The recorded load was reproduced in the testing environment in several phases, with various configuration parameters of the mass storage environment, server platform and network.
“Over several months, we tested, step by step, every parameter included in the Hitachi proposal. We took the declared values very literally and we were very meticulous,” recalls Księżycki. “In addition, every testing scenario was run for different variants in terms of configuration and resources available in each layer of the environment, for different volumes of jobs being executed simultaneously, for a variety of simulated problems, etc. All tests were planned and documented so we could repeat them or compare the results when needed.”