Solution import error with environment variables in deployment settings file

We recently had a funky issue where we kept getting the following error when we tried to import a solution in our upstream environments:

The reason given was: The solution import failed because some references included in the solution are not present in the organization. List of unresolved references: 'environmentvariabledefinitionid - <schemaname>mal_advisor</schemaname>', 'environmentvariabledefinitionid - <schemaname>mal_emergency</schemaname>', 'environmentvariabledefinitionid - <schemaname>mal_hotline</schemaname>'

These are environment variables that we had used for a period, but was deleted some time ago. I proceeded by checking the following things:

  1. I checked our dev environment for these environment variables, confirmed that they did not exist anymore
  2. Checked our upstream environments (test, prod, etc), but they weren't there either
  3. Downloaded solution zip file and extracted all contents, but couldn't find a mention of any of the variables in any of the files
  4. Tried importing the solution manually, and that worked.

So that made me think the issue must be in our pipeline somewhere, causing the import to look for variables that don't exist. The most obvious thing (to me) would be the deployment settings file, and sure enough it turns out our build pipeline didn't regenerate the deployment settings file properly. I removed the variables from the deploymentsettings.json and ran the pipeline again, and this time it worked as expected.

PS: In case you don't know what deployment settings are, here's a Microsoft learn article explaining it and also how to generate one: https://learn.microsoft.com/power-platform/alm/conn-ref-env-variables-build-tools