Friday

Unlocking the Power of Open-Source Intelligence for a Data-Driven Army

By Colonel Christopher Tomlinson, Chief Warrant Officer 3 Felix Rodriguez Faica, Angela White, And Kathryn Ruhl

Article published on: July 1, 2025 in July-December 2025 Issue" to "in the July – December 2025 Semiannual Collection

Read Time: < 5 mins

Introduction

The modern intelligence landscape is characterized by unprecedented opportunities and challenges. The sheer volume of publicly available information (PAI) and open-source intelligence (OSINT) offers invaluable insights into emerging threats and complex operational environments where collection assets are limited. Effectively harnessing PAI and OSINT at speed and scale requires overcoming significant hurdles, particularly when integrating unclassified information into classified intelligence workflows. To take full advantage of OSINT and enable near-real-time intelligence analysis, the U.S. Army must prioritize both technical interoperability and policy reform to streamline the flow of open-source data into classified analysis.

The FRIDAY project, developed by Southern European Task Force – Africa’s (SETAF-AF) Africa Data Science Center (ADSC), enables the seamless and secure movement of OSINT data from the unclassified Non-Secure Internet Protocol Router Network (NIPRNET) to the classified Secret Internet Protocol Router Network (SIPRNET), where analysts can use the data to enhance object-based intelligence production within existing enterprise systems like the Army Intelligence Data Platform (AIDP). Produced through a collaboration between military personnel and civilian data scientists, FRIDAY utilizes a novel data processing capability to overcome interoperability limitations in current intelligence programs of record, enabling a holistic and data-driven approach to intelligence analysis. FRIDAY addresses the critical need for rapid and timely conversion of open-source data into actionable intelligence on classified systems. SETAF-AF is thus empowered to capitalize fully on the wealth of information available in the open-source environment, which ultimately strengthens the overall security posture within its area of responsibility.

Currently, turning raw OSINT data into actionable intelligence objects within AIDP involves a series of multiple handoffs between different teams and systems. This fragmented approach risks creating bottlenecks, increases the potential for errors, and limits the speed and agility of the intelligence cycle. FRIDAY tackles the fragmented multi-domain challenge head-on through a streamlined process using existing government off-the-shelf (GOTS) and commercial off-the-shelf (COTS) systems that are readily accessible to the Army intelligence community. Instead of relying on cumbersome and error-prone manual creation and re-creation, FRIDAY automatically and securely moves OSINT reports from NIPRNET to SIPRNET, eliminating a significant workflow bottleneck for analysts. This not only frees up bandwidth for analysts but also ensures that analysts operating within classified environments have ready access to valuable OSINT insights. Additionally, the FRIDAY project addresses the critical need for secure information sharing between different classification domains using existing enterprise tools with defense and intelligence community standard authentication methods and group- and role-based access controls.

As shown in the figure above, FRIDAY is not just a tool but a pipeline that features a user interface for data entry, an environment for data processing, a cross-domain solution, and, at the final step, integration with AIDP. Recognizing that OSINT input often comes in inconsistent formats, FRIDAY implements a crucial step: data normalization. As the OSINT collector enters OSINT reports, FRIDAY standardizes their format. Ensuring data consistency and ontology compatibility regardless of the original source or structure makes the data resistant to “anomalous usage patterns found in intel traffic.”¹ Data normalization is a requirement for seamless integration with enterprise tools like AIDP to allow OSINT data to corroborate other sources for intelligence purposes.

Figure. FRIDAY workflow

Once FRIDAY processes and transfers OSINT data to the SIPRNET environment, it undergoes a crucial transformation into “ontology objects” within AIDP. These objects represent key entities, events, and units extracted from the OSINT reports, and they enrich the existing intelligence picture with valuable insights gleaned from the open-source realm. This object-based approach goes beyond simply adding more data; it connects the dots between seemingly disparate pieces of information. By linking OSINT-derived objects with classified data already present in AIDP, analysts achieve a more comprehensive understanding of the operational environment.

During design and development of the pipeline, ADSC conducted tradeoff analyses and testing with multiple technologies to prototype a solution. Advana, Chat Surfer, and Microsoft Power Platform are already acquired GOTS and COTS systems, allowing ADSC to bring the tool from design to user-acceptance testing in under four months with zero added cost to the organization. While the current implementation (outlined in solid black in the figure referenced by the status indicator) balances robustness and feasibility given the availability of the tools, further iteration and analysis are required to attain long-term viability and scalability.

Ultimately, the goal is to establish near real-time connectivity between open-source information and classified analysis. To unlock FRIDAY’s full potential, the U.S. Army must break down the barriers to true interoperability, which present in two categories: technical and procedural. Technical interoperability requires compatible schema definitions, practical ontologies, data governance and security best practices, and avoiding vendor lock-in. Procedural interoperability is sometimes more difficult to achieve. It requires different organizations with idiosyncratic, people-driven processes to design systems using common or compatible technical specifications and to embrace VAULTIS data standards,² often entailing daunting cultural shifts on top of technical project setup. Further, organizations must designate stewards to take responsibility for data initiatives beyond initial operating capability and into the maintenance phase.

Compatibility issues in integrating data science and engineering (DS&E) tools with, for example, legacy systems present data formatting discrepancies and security challenges that hinder the smooth exchange of information between new and existing systems. Addressing these interoperability hurdles requires a strategic approach that identifies technical and procedural limitations and deliberately weighs the costs of long-term solutions against the risks and opportunities of short-term workarounds.

While the benefits of integrating DS&E into intelligence workflows are undeniable, we must continue to highlight additional challenges to fully realize its transformative potential. One hurdle is overcoming cultural resistance to new technologies and approaches. Many intelligence professionals steeped in traditional methods may be hesitant to embrace DS&E, perceiving it as disruptive or overly complex. Therefore, fostering a deeper understanding of DS&E capabilities among both analysts and leadership is crucial.³ This requires demonstrating the tangible value of DS&E through concrete examples and success stories, highlighting its ability to enhance, not replace, existing expertise.

Conclusion

Finally, building a sustainable pipeline of skilled data science professionals is paramount for long-term success. This requires a multifaceted approach that encompasses targeted training programs for existing intelligence personnel, recruitment efforts aimed at attracting top data science talent, and the establishment of career paths that recognize and reward expertise in both intelligence and DS&E. By investing in workforce development, the intelligence community can ensure it has the skilled personnel necessary to leverage the power of data science effectively for years to come.

Initiatives like FRIDAY demonstrate the transformative potential of DS&E in modernizing intelligence operations, particularly in leveraging the power of OSINT. By examining the factors that inhibit innovation within the enterprise, and encouraging data-driven solutions, the U.S. Army can maintain its strategic advantage in the face of evolving threats and complex operational environments. Continued investment in DS&E infrastructure, training, and research will be critical for ensuring timely, insightful, and actionable intelligence reaches decision-makers at all levels.

Notes