BEHIND THE NUMBERS: The FBI Cyber-Crime Survey Results
Carter Schoenberg questions the accuracy of the 2004 CSI/FBI Computer Crime and Security Survey. “The trends suggested by data in the report are not consistent with the trends we are seeing,” says Schoenberg, a senior analyst in the X-Force threat intelligence unit of Internet Security Systems Inc. (ISS), an Atlanta-based provider of products and services to protect computer networks against Internet threats.
The San Francisco-based Computer Security Institute (CSI) and the San Francisco Federal Bureau of Investigation’s Computer Intrusion Squad have conducted the CSI/FBI survey annually since 1996. The 2004 effort polled 494 computer security professionals drawn from corporate, government and institutional sources.
Respondents came from financial, high-tech and manufacturing businesses as well as government agencies and educational institutions. Thirteen percent of respondents came from government organizations, while 19 percent were designated as coming from “other” groups. People from small, medium and large organizations were polled for the survey, with the largest pool of respondents coming from organizations with 1,500 to 9,999 employees.
The survey, released in late May, reported a decline in financial losses caused by security breaches compared to the year before. The survey also reported that the most expensive computer crime is now denial-of-service attacks. Previous surveys had ranked intellectual property theft as the most expensive.
Schoenberg’s wariness about the survey’s findings begins with a critique of the survey group selected by CSI/FBI researchers. He contends that nothing in the survey indicates whether or not the 2004 respondents are the same people as those polled last year. “If you don’t talk to the same respondents every year for this kind of survey, how can you establish baselines for making judgments about trends?” he asks. According to Robert Richardson, editorial director of CSI and a co-author of the report, respondents do not vary significantly from year to year. “CSI is a membership organization, and we poll our members,” he says. “It’s certainly possible that some people respond one year and not the next, but we are essentially polling the same group. It’s also true that we do not do longitudinal tracking, the statistical procedure that helps you make sure that you are talking to the same people every year. We’ve made a conscious decision to keep our surveys anonymous” in order to ensure response to questions about financial information.
Schoenburg has a bigger quarrel with the survey’s financial loss findings, which show that losses decreased to $141.5 million in 2004 from $202 million in 2003. Schoenberg points out that these numbers are dramatically lower than those found by a U.S. Secret Service survey that pegged losses of $666 million from cyber-crime last year. In addition, the Secret Service survey also indicates an increase — not a decrease — in financial losses related to cyber-crime in the most recent period.
“The Secret Service numbers are much more in line with other surveys I’ve seen and with my own experience,” Schoenberg says. Richardson attributes the different findings to different survey groups. “Our members are definitely a biased group,” he says. “They have joined CSI because they want to find ways to reduce economic losses. If we’re doing anything right at all in the security work connected to CSI, it isn’t shocking to see lower loss numbers or declines in losses reported by these respondents.”
Finally, Schoenberg questions findings related to the most expensive computer crimes. The CSI/FBI survey contends that denial-of-service attacks overtook theft of intellectual property as the most costly computer crime.
The graph in the report that summarizes losses shows that denial-of-service attacks cost about $26 million in 2004, compared to $11.5 million in losses due to the theft of intellectual property. Richardson calls this a legitimate observation. “We made an error,” he says. “The loss summary data does not include losses caused by viruses, and it should. Our respondents reported that viruses caused losses of $55 million last year, more than twice the total for reported denial-of-service losses. So viruses, and not denial-of-service attacks, were the most costly computer crime noted by our members.”
While Schoenberg disputes some of the findings in the CSI/FBI survey, he also says that the survey has its uses. “I think the report is an accurate representation of the data collected,” he says. “The problem is that you can’t use it to define larger trends in cyber-security. You really can’t take any survey information at face value. You have to corroborate it.”
