Continuous process improvement at eBay. The quest for automation leads to optimization.
"Serena's (now part of Micro Focus) help was critical and you guys were very responsive. We had daily meetings and what kept me going was your commitment to making us successful."
Senior Change Manager
At a Glance
- Increased scrutiny and reduced the time to production
- Reduced significant amounts of rework and lost business productivity
- Boosted user confidence with IT
When your business is creating the ecommerce platform for thousands of other companies and hundreds of thousands of individuals, a single system change can have profound consequences. At eBay, millions of dollars’ worth of transactions are successfully completed every day so software change management has to be a discipline they excel at.
When GSI Commerce was acquired by eBay in 2011, two technology cultures met and set about creating a modern software development lifecycle able to meet the challenges of one of the fastest moving and innovative companies in the market. GSI Commerce’s evolution was that of a startup dedicated to being responsive to every customer and determined to move fast and worry about breaking things later. “In many ways,” says Setlock, “GSI were a victim of their own success. Time to market was everything.” eBay’s heritage spoke to cautious financial stability at the expense of some velocity in deploying innovations.
Rick was brought into eBay because of his deep domain expertise and experience in software change management forged at Merrill Lynch and at Educational Testing Services.
His first task was to align practices, processes and procedures with corporate policies and programs. eBay’s senior management were very supportive of this and encouraged him to look at technology solutions to support his efforts. Executives were very knowledgeable about the business and clearly understood the risk reduction and competitive advantages that come from having a secure change management process.
Rick knew that the way to be effective quickly was to choose those processes that were causing the most pain. Even though there were tools in place they were not being used as intended and had such low adoption rate as to make them ineffective at best and actually harmful at worst.
In one example Micro Focus Dimensions CM, the leading software change, configuration and release management solution, was being used as an approval tool just because of its built-in workflow. But it wasn’t being used by everyone and over 15,000 tickets were in “open” status in the system even though those changes had long since made it into production.
As is so often the case, email, spreadsheets and paper were holding the process together. Lack of knowledge of how the tools worked resulted in significant amounts of manual effort especially around reporting.
“Simplify and streamline”, Rick told the team. “Let’s make our job easier to understand and easier to do for us, and that’ll make it easier for everyone else.”
Taking the Change Management process, he eliminated all the duplicate states, simplified every decision point into a binary choice and ensured that no change went into production unless it came through the process. Adoption rate was actually very good. Everyone recognized that the process changes simplified the CR workflow and made it easier to implement changes.
The Change Request process, as eBay calls it, was very simple but very effective. If you want to release code, you need to make a formal Change Request. If you want to add code to a release you need to submit a request for that too. As these requests got approved, notifications to all stakeholders were sent out so that everyone knew what was coming. No request— no deployment. No approval—no deployment.
The results were impressive. All requests were routed through the entire workflow which included the steps to deploy and validate in the production environment. As a result, the approval time decreased and the number of open change requests decreased from 15,000 to less than 50 at any given time.
With this immediate success other teams along the software development lifecycle (SDLC) began to take notice. Other development teams began to request additional processes to assist their development efforts.
Their success is tied to their purpose—creating more opportunities and enabling others to win while making a positive social impact.
With 157 million active buyers globally, eBay is one of the world’s largest online marketplaces, where practically anyone can buy and sell practically anything. Founded in 1995, eBay connects a diverse and passionate community of individual buyers and sellers, as well as small businesses. Their collective impact on ecommerce is staggering, and approximately 800 million items are listed on eBay.
Seeing the momentum shift Rick realized that this was the perfect time to put end-to-end process control into the SDLC. Executive management were supportive and willing to fund the project. The dev change, release and ops teams were impressed by what had been done thus far and the Dimensions CM technology was due to be upgraded.
It was the perfect time to extend Change Management further back along the SDLC. Then one day, as Rick was designing his process, the eBay issue and defect management system failed and the entire MS Access database was lost. The Access DB failure was what initially prompted eBay to purchase a new IDM (Issue and Defect Management) system. They chose CA Service Desk as it was already in use at eBay and, after 6 months of work, it was clear that it was the wrong tool for them. It was simply too costly to implement, difficult to maintain and did not meet the business requirements. Worst of all it didn’t integrate with other CM process applications.
Senior executive leadership at eBay realized that depending on a homegrown IDM system based on MS Access was not going to give them infrastructure that would scale. Add to that tools that were hard to use and hard to update for the wider service management processes.
Rick was tasked with selecting a solution for the whole of ITSM at eBay. He looked at Service Now and Remedy (already in use at eBay) but found these products to be very expensive and quite inflexible.
That’s when Rick turned to Micro Focus to see how its solution stacked up. When he saw SBM-based Service Support Manager for the first time, he was impressed by the speed at which he could turn process documentation into executing online processes. “The way you guys set up the initial workflows gave us everything we need”, commented Rick after the initial pilot of Service Support Manager.
Service Support Manager, an out of the box ITIL-verified, set of integrated workflows for problem, incident, change, release, service desk, knowledge base and financial asset process, seemed like the best way to get best practices across all parts of IT as well as operational workflows implemented and benefiting eBay in the shortest time.
The initial project was to roll Solutions Business Manager out product by product. “We converted all 75 products in just six months and we got zero negative feedback,” said Rick.
One huge saving was found in the Change Control Board (CCB) meeting. At this meeting, which happened three times per week, every proposed change was scrutinized and approved for deployment. “You might be in the meeting for just your one change that might not be discussed until the end of the session,” Setlock observed when talking about the immediate savings created by this project. “The meeting usually lasted up to an hour, often ran over, and had 30 to 40 people in attendance. The annual cost of the time spent by employees in the meeting alone represents over $250,000 a year.” He added, “we completely eliminated the meeting and now you only have to worry about your own changes."
One other frustration also went away with the new workflow automation delivered by SBM. It used to be that if your change missed the CCB meeting you had to wait until the next one for approval. If you wanted to rush it through you had to go around the office getting approvals to force through the emergency change. “Now your normal change can glide through the workflow faster than the old emergency way of doing things, the CCB is online 24- by-7,” Setlock proudly commented and added, “productivity went through the roof, and we increased scrutiny and reduced the time to production.”
A critically important change was to force the QA signoff ahead of the deployment approvals. In the past, the CCB (Change Control Board) was approving changes before QA had certified the code. Now, because of the inherent speed in the process, eBay is able to enforce QA acceptance every time before delivery of the code to production. This impact alone has saved significant amounts of rework, improved business productivity and boosted user confidence in ITs ability to deliver the solutions they need.