Australia needs an open-source intelligence agency

Because people have been spying for most history, it’s easy to forget that our intelligence bureaucracy was developed primarily to fight the Cold War. After the Cold War the digital revolution took off. The consequent transformation of the information environment is the most important development in the history of intelligence, at least since the Second World War.

Australia must adapt its National Intelligence Community (NIC) to information age. It’s true that Australia and its Five Eyes partners have reformed the intelligence institutions established in the Cold War. But the lodestar of reform for the last two decades has been counterterrorism. The independent review of NIC, which will submit its report in 2024, should focus more directly than its predecessors did on the challenges of information age.  

The sheer volume of information is its most striking feature. Turning torrents of data into useful information increasingly requires specialist skills. The disciplines of collation, verification, curation, and analysis have come to be known as open-source intelligence (OSINT). OSINT organisations, ranging from commercial data brokers to investigative NGOs such as Bellingcat are producing more intelligence that rivals that of secret intelligence organisations.

But OSINT won’t replace secret intelligence. Australia needs – perhaps more than ever – to understand the secret intentions and capabilities of its competitors and adversaries. The war in Ukraine has demonstrated the value of both forms of intelligence. OSINT has pierced the fog of war while, the United States and United Kingdom used secret intelligence before Russia’s invasion to alert allies and “pre-bunk” Russian disinformation.

The real challenge is to balance and integrate the production of OSINT and secret intelligence. OSINT should provide foundational intelligence. Scarce classified resources to be focussed on hardest targets and most valuable secrets. Secret intelligence should be used to direct, refine and validate OSINT. Quality OSINT can be shared with allies and partners far more easily than secret intelligence because the risk to sources and methods is greatly reduced.

Still, integration remains a challenge because OSINT and secret intelligence differ in fundamental ways. According to intelligence expert Amy Zegart, they are different “ecosystems”. Each discipline has distinct advantages and disadvantages. OSINT is cheaper and faster. Secret intelligence is more tailored and revelatory. A degree of opacity is necessary for secret intelligence while transparency usually improves the quality of OSINT.

Integration remains a challenge (Rosie Steggles/Unsplash)

But secrecy comes at a cost. Secret intelligence organisations tend to over-classify and under share information. They generate a culture secrecy which views information as valuable because it is secret rather than vice versa. This culture is at odds with the new information environment and frequently invoked as the main obstacle to the adoption of OSINT by secret intelligence organisations.

Australian intelligence organisations are making more and smarter use of OSINT. But the NIC remains geared towards secret intelligence. That’s not surprising. Australian intelligence was, after all, established as part of a multi-layered Five Eyes bureaucracy designed to collect, protect, analyse and disseminate secrets. That bureaucracy delivered important wins in the Cold War and the war on terrorism, but a major shift towards OSINT is now necessary.

Australia should establish a dedicated OSINT organisation within the NIC. This would recognise – and make the most of – the fundamental differences between OSINT and secret intelligence. Housing the OSINT agency within the portfolio of Foreign Affairs and Trade would better connect it to the vast but under-utilised pool of knowledge contained in DFAT’s international network.

Done right, this reform could also disrupt the culture of excessive secrecy. The integration of OSINT and secret intelligence should not preclude healthy competition between the two disciplines. Secret intelligence is already competing with unverified social media feeds for the attention of ministers. A more managed competition between OSINT and secret intelligence should be harnessed to improve the overall quality of intelligence, regardless of its source.

Competitive pressure from OSINT should encourage the NIC to focus on valuable secrets, to correctly classify information and to prioritise the usability of intelligence. The downgrade, declassification and dissemination of intelligence by the United States and United Kingdom before and during Russia’s invasion of Ukraine has been rightly hailed as a success. But it is also true that much of this intelligence was, therefore, originally over-classified.

National security in the information era will depend more on the ability of states to obtain, protect and use data. But the new information environment is still taking shape. New technologies are both clarifying and confusing. We still have much to learn about the complex dynamics of information competition virality and disinformation, as well as how these dynamics shape human perception, cognition and action.

So, the NIC must, above all, become more flexible. Already, the distinction between OSINT and secret intelligence is starting to blur as cyberspace is fragmented by digital sovereignty, firewalls, and ubiquitous encryption. It’s distinctly possible that what we now know as secret intelligence will one day be a niche specialisation within the larger OSINT enterprise. Establishing a dedicated OSINT organisation would not cause such a radical shift. But it would better prepare Australia for that, and other, possibilities.

Ben Scott is author of Adapting Australian intelligence to the information age, an Occasional Paper published this week by ANU National Security College.