Fresno County pays IBM $2.3 million following software audit

Fresno County paid IBM almost $2.3 million after a software-licensing audit revealed too many workers were using unlicensed IBM software in county departments.

The bill, which was paid in March, is leading to changes in the way the county buys and uses software.

County officials say the bill could have been much worse.

“We got a hell of a deal,” Board of Supervisors’ Chairman Buddy Mendes said.

IBM initially sought nearly $4.5 million to settle the audit claim, but the county negotiated to get the bill lowered. In some cases, software was on computers of long-retired employees and never had been removed, said Robert Bash, the county’s chief information officer.

Other costs stemmed from a misunderstanding of the contract, he said.

After supervisors approved the payment, an anonymous letter transmitted through internal county emails suggested the county’s software issue stemmed from management problems in the internal services department.

$2.3 millionAmount paid by Fresno County to IBM after software licensing audit

Mendes credits Bash for lowering the payment and bringing the years-long problem to supervisors’ attention.

“It was the system’s fault,” Mendes said. “That was being done for years.”

Supervisors originally approved a contract in July 2014 that paid for $5 million in software, associated maintenance and customer support for five years. In early 2015, IBM paid for a third-party audit to determine how Fresno County was using its software, which uncovered the unlicensed software users.

IBM’s audit covered one year in the past and one year into the future assuming similar levels of use. The nearly $2.3 million will pay for software licensing and maintenance for those years.

Such audits are fairly routine and not easy to fight, said Fresno County Counsel Dan Cederborg.

He said the county weighed its options, but legal action against IBM wasn’t advised because the corporation is responsible for confirming its number of licenses and charging accordingly.

“Obviously, we looked at all our options, but that wasn’t one of the viable ones,” he said. “I think the (Internal Services) department did a good job in working down the number.”

Longtime issue

Some software misuse issues, Bash said, date back to the late 1990s, although they weren’t part of the audit. Bash was named chief information officer in early 2014.

The most significantly misused software program was in the county’s Department of Social Services, county documents said.

Under the settlement, the county pays for new licensing and back maintenance for software overuse, he said. In other cases, software hadn’t been used for long periods of time because current and former employees retained software on their personal computers but hadn’t deleted it, Bash said.

Another reason the payment dropped was because IBM wanted to charge the county for having more than 7,000 employees with access to certain software when only one department had most of the unlicensed software installed, Bash said.

“We were supposed to do software reconciliations as we went along, and that wasn’t happening,” he said. “It has to happen on a regular basis, and you have to pay for what you deploy.”

County officials thought the contract was supposed to allow a certain number of users on software at one time, but the contract was more stringent.

We were supposed to do software reconciliations as we went along, and that wasn’t happening.

Robert Bash, Fresno County’s chief information officer

“There was a misunderstanding of what the county’s license allowed,” Bash said. “It was based on some representations made by their people, but there were some things that were not done right in the past as well.”

Supervisor Debbie Poochigian, the board’s chairwoman last year, said she recalled discussions about allowing a certain number of people to use software at the same time.

“We spent a lot of time negotiating” those connections, she said.

Most of the $2.3 million settlement will be paid out of state and federal funds because the state and federal governments pay most costs for Department of Social Services’ operations. The general fund cost was $6,762.

Bash’s office also will need $286,015 to cover the department’s costs for IBM’s findings on other software products.

Other costs will be paid by the Information Technology Services Division to administer a program to acquire software for county departments. Those costs will be reimbursed by individual departments that order the software, he said.

“It’s bringing back the responsibility to us,” he said. Information Technology “should have been wholly responsible.”

Tighter controls

Texas-based technology lawyer Julie Machal-Fulks said IBM has been stepping up its audits, which more often than not take on an adversarial tone.

“Their position is that everyone should always be prepared for an audit,” Machal-Fulks said. “They are pretty forceful  pretty adamant about enforcing their rights.”

Fresno County’s settlement isn’t unusual, Machal-Fulks said.

“I’ve seen cases from companies that are told they owe a few hundred thousand to 40 or 50 million,” she said.

An IBM official did not return a call for a comment.

I’ve seen cases from companies that are told they owe a few hundred thousand to 40 or 50 million.

Julie Machal-Fulks, technology lawyer

Board Chairman Mendes said the county is taking responsibility for failing to police software licensing and use by employees.

“There was no administrator taking care of the software, and people were grabbing it because nobody had a lockdown on who could have what,” he said. “The system was screwed up.”

Poochigian said the county should have tighter controls on software and monitor such issues as when people retire with county software on their computers.

“That’s a lot of money, and we have to make sure the people responsible are watching for these kinds of things,” she said.

Bash said a more centralized process already is in place. All software purchases are run through his office. An invoice then goes to the department where the software was ordered for payment.

The county also will start doing its own software licensing audits, he said.

“This will reduce the ability for departments to install their own software so we don’t end up overdeployed,” Bash said.

Marc Benjamin: 559-441-6166, @beebenjamin