Building a Resilient Strategy Against Algorithm Changes