The last time I saw this, we debated hard around this fixation on running third-party admin-level agents on boxes. Especially since our side was running non-windows, we maintained that our setup adequately mitigated any issues. They were adamant, and the contractors they brought in were boneheads, and we were not friends.
But. The brass said capitulate. Our terms:
they acknowledge they’re asking for unknown ‘black box’ third-party software into hosts to run with admin-level access. They hammered at that one hard but it’s not wrong.
they acknowledge that it poses an avoidable risk and it’s done entirely at their express and direct request.
they furnish a bond in an amount to cover the rebuild of the related hosts because it’s a risk.
with their link to the rest of the world, these hosts are declared non-sovereign, and no P-I can be near them.
the hosts live in a ring-fence to protect the rest of the organization from their non-sovereign selves
they furnish a team and a member standing-by like a regular stand-by team to respond to alerts related to potentially-spotted viruses on the hosts - even if not immediately considered a risk to the hosts.
they sign indemnity agreements stating this was all their decision, that we had consulted them about increased costs and reduced effectiveness, and risk, and they understood and accepted it.
They signed readily.
I think that was it. This isn’t my plan, since I’m just not that smart, but the guy who itemized the terms was really good.
So we got machines, ring-fenced them, locked them away and got the scanner agent set up. Every day or so we’d have our stby wake their guys up - and they were bargain contractors so, no standby pay - and they’d get to go over an analysis of the suspected virus, decide it was nothing - suspected no risk, but inconclusive (no one says 100% in security) - and go back to bed.
This went on for weeks; even after the machines were deemed unusable for the project. You see, it was a system that handled registrations … for something; names, numbers, times; P-I. And they couldn’t handle P-I because they were non-sovereign and we couldn’t violate our data sovereignty requirements.
But we had to set it up. Someone paid good money for that hardware; which, because it had to be ring-fenced, couldn’t be part of our standard private-cloud (on prem) setup. Wow, but that budget went out the window.
After only a few weeks of this comedy they begged for a meeting. “Kill the agents,” they said. But why? We paid for them and it’s in the plan. You agreed to support the agents for the life of the project!". Okay, we laid it on thick.
They said please no; they’d go with our plan. “our plan? We like your plan. You want to do this other thing, we’re gonna need indemnity for the risks in the plan you call ours that you rejected but now you want to do.” And they could really only agree.
Agents removed. Boxes rebuilt - on their dime because of the “inconclusive” above and the “bond” clause - as we can’t be sure what went on those boxes. We fulfilled the VM requirements from our genpop, and added the new boxes in as common hardware as the hardware was EOSL and the maker wanted to support it but never see it again. They took a penalty because what we called their bumbling ran them over the delivery timeline, but at least it was done. Project complete, brass pats themselves on the back, cheapo contractors stung and tired. Kinda less our friends at that point…
And we got new hardware, which the CFO said was never gonna happen. Like a half-mil he signed-off on, and that went into genpop as mentioned. And every time they went to hire a cheap whore, we got to remind them of how this one ran over so poorly, and they get to explain themselves to the top brass. That was a gift that kept on giving, even if THEY reminded us how much the contractors didn’t earn via delivery penalties and how much they had to pay back on that “bond” clause, which was just labour we needed to spend anyway on fixing a delivery process.
And we laughed like the end of a Bellisarius serial right before the freeze-frame.
I think the security researcher has a valid point.
In a secure environment you don’t want random things running in memory, sending samples to third parties.
Would a static virus scanner run periodically on the volume itself been sufficient? If yes, then the researcher was being unreasonable.
The last time I saw this, we debated hard around this fixation on running third-party admin-level agents on boxes. Especially since our side was running non-windows, we maintained that our setup adequately mitigated any issues. They were adamant, and the contractors they brought in were boneheads, and we were not friends.
But. The brass said capitulate. Our terms:
They signed readily.
I think that was it. This isn’t my plan, since I’m just not that smart, but the guy who itemized the terms was really good.
So we got machines, ring-fenced them, locked them away and got the scanner agent set up. Every day or so we’d have our stby wake their guys up - and they were bargain contractors so, no standby pay - and they’d get to go over an analysis of the suspected virus, decide it was nothing - suspected no risk, but inconclusive (no one says 100% in security) - and go back to bed.
This went on for weeks; even after the machines were deemed unusable for the project. You see, it was a system that handled registrations … for something; names, numbers, times; P-I. And they couldn’t handle P-I because they were non-sovereign and we couldn’t violate our data sovereignty requirements.
But we had to set it up. Someone paid good money for that hardware; which, because it had to be ring-fenced, couldn’t be part of our standard private-cloud (on prem) setup. Wow, but that budget went out the window.
After only a few weeks of this comedy they begged for a meeting. “Kill the agents,” they said. But why? We paid for them and it’s in the plan. You agreed to support the agents for the life of the project!". Okay, we laid it on thick.
They said please no; they’d go with our plan. “our plan? We like your plan. You want to do this other thing, we’re gonna need indemnity for the risks in the plan you call ours that you rejected but now you want to do.” And they could really only agree.
Agents removed. Boxes rebuilt - on their dime because of the “inconclusive” above and the “bond” clause - as we can’t be sure what went on those boxes. We fulfilled the VM requirements from our genpop, and added the new boxes in as common hardware as the hardware was EOSL and the maker wanted to support it but never see it again. They took a penalty because what we called their bumbling ran them over the delivery timeline, but at least it was done. Project complete, brass pats themselves on the back, cheapo contractors stung and tired. Kinda less our friends at that point…
And we got new hardware, which the CFO said was never gonna happen. Like a half-mil he signed-off on, and that went into genpop as mentioned. And every time they went to hire a cheap whore, we got to remind them of how this one ran over so poorly, and they get to explain themselves to the top brass. That was a gift that kept on giving, even if THEY reminded us how much the contractors didn’t earn via delivery penalties and how much they had to pay back on that “bond” clause, which was just labour we needed to spend anyway on fixing a delivery process.
And we laughed like the end of a Bellisarius serial right before the freeze-frame.
That’s a real roller coaster ride of a journey. Thanks for sharing it. Glad you got some bonus hardware out of it.
I’d like to say it was all masterfully done plan, but I’m sure it was 90% CYA and luck.
Still, yeah, hardware budget was non-zero, and when FIN is pulling the strings, that’s always nice. ;-)
Totally reasonable to not do a dumb thing if you have no contractual obligation to do the dumb thing.
Sadly they had that obligation, so they have to weigh the cost of doing the dumb thing with the cost of breaching contract.