It’s a matter of speed – Best in Bags does a really thorough examination of all of the combinations of gear in your bags, of which there can easily be trillions. It takes a couple seconds to to this – for the sake of argument here, let’s say it takes exactly 2 seconds.
With the upgrade finder, we are ranking potentially hundreds of items for you at once. To guarantee the same exact result as Best in Bags for each one of those items, we would have to put it in your inventory, run Best in Bags, save the result. Repeat for the next item in the list, and so on. If the list has 100 items, that would take 100 times 2 seconds per optimization, or a little over 3 minutes.
I don’t think that most people would want to sit there for 3 full minutes waiting for an answer… not to mention how expensive that would be to run, which would significantly raise the price we need to charge users for the website.
Instead we use a quicker ranking method for the upgrade finder that gives you a pretty close approximation of how valuable an item would be were you to get it right now. The idea is that even if you get it and run Best in Bags right now it is equivalent to your current gear or even a slight downgrade, it’s very possible that as you get more gear, it will become worth using.
While your case certainly does happen, it is actually pretty uncommon. The majority of the time, if the Upgrade Finder says it is an upgrade, Best in Bags will choose it. This generally only happens when the upgrade is really small.