Challenges and Lessons Learned in Data Integration

‘Complex but necessary’

By Capt. Brian Lee Matias, 4th Combat Aviation Brigade

Article published on: April 1, 2025, in the Army Communicator Spring 2025 Edition

Read Time: < 6 mins

Capt. Brian Lee Matias, 4th Combat Aviation Brigade

The Army's need for modernization is undeniable, particularly in the Digital Age, where data promises to transform military operations. Leaders across the globe in all types of units are tasked with exploring how to leverage data, integrate it into operations, and fully utilize its potential. However, the path to data integration has not been smooth.

The 4th Combat Aviation Brigade (CAB) has spearheaded efforts to harness data, enabling leaders within the unit to make rapid decisions based on the most accurate and timely information available.

Importance of Data Awareness

At its outset, the Army's journey toward data integration faced a critical issue: misunderstanding the concept of data. Leaders were mandated to implement data-driven solutions without fully grasping what data entails or how it could be effectively used.

According to Cambridge University, data refers to facts or numbers collected to assist decision-making. In the Army, this data empowers leaders to make informed, rapid decisions under pressure. This lack of clarity in defining data across units led to its misuse in many contexts. For example, some leaders referred to routine reports or isolated anecdotes as "data," failing to distinguish between raw information and actionable, collected insights meant for strategic decision-making.

The Army recognized the need for modernization in various areas, including Soldier performance monitoring to equipment maintenance, supply chain optimization, and training enhancement through predictive analytics. However, the leap from recognition to implementation was fraught with difficulties.

Struggles with Data Integration

A significant challenge we faced in integrating data was the Army's reliance on outdated tools, like Excel, which are inadequate for handling large datasets and complex analyses. Our data team within 4th CAB consisted of four members, each with diverse expertise, working together to overcome these obstacles. While more advanced tools such as Power BI offered promise, integrating these tools with existing Army systems was extremely difficult.

One key hurdle was the Army's IT infrastructure, managed by the Regional Network Enterprise Center (RNEC). Due to cybersecurity restrictions and administrative roadblocks, requests for installing essential tools like Python and Open Database Connectivity drivers were often delayed or denied, stifling our ability to rapidly adopt new tools for data analytics effectively.

Learning from the Ground Up

When 4th CAB established its data team, the team members were selected based on minimal experience or mere interest in the field. Many had no experience using data analytic tools. This led to a steep learning curve where team members had to teach themselves how to use the tools, often relying on external resources like YouTube tutorials.

Unfortunately, many examples found in these resources did not align with the unique challenges faced within military operations, mainly due to the sensitive nature of military data and a plethora of disparate authoritative sources. To bridge the knowledge gap, we supplemented our learning with Udemy courses on data integrity, connection, and Power BI usage. Additionally, some team members pursued the CompTIA Data+ certification, hoping it would provide further expertise in managing and analyzing military data. As the data team grew in capability, it became evident that managing data systems could not be an additional duty. The time-intensive nature of the work, coupled with the complexity of the systems involved, required a dedicated role. Without full-time personnel responsible for data integrity and maintenance, we often came in early or stayed late to resolve issues.

Overcoming Fragmentation

One of the core struggles of implementing data in the Army was the decentralization of information. Every section, every duty position, and every leader approached data differently. They utilized different systems of record and had their own methods for pulling and analyzing data. This fragmented approach created a lack of synchronization and standardization. For data to be effectively integrated, we had to centralize the information and establish a uniform method for data pulls across the entire organization. This process required extensive coordination and a clear visualization standard for presenting data to leadership.

From Chaos to Standardization

The widespread adoption of Power BI was a turning point, but it came with challenges. Without any formal training or guidance, each section was left to figure out how to use the tool on their own. Instead of a coordinated effort, the team relied heavily on one or two self-taught experts to execute critical tasks and guide others through the complexities of the platform. The lack of a clear goal for what we were trying to achieve compounded the problem. Leaders wanted data presented in a way that allowed for rapid decision-making, but no one knew how to get there.

After much trial and error, we finally established data visualization and reporting standards. However, the difficulties didn't stop there. In many cases, we couldn't pull accurate data directly from systems of record and manually compile reports using Excel, which was time-consuming and prone to error. As a result, we had to refine our processes continually to ensure data accuracy and relevance.

Overcoming RNEC Limitations

A significant roadblock we encountered was the limitations set by the RNEC. While we eventually received Power BI Pro licenses, many of the tool’s advanced capabilities – such as Python scripting and database integration with systems like SQL servers and Amazon Redshift – remained inaccessible due to strict cybersecurity regulations. These constraints hindered our ability to fully exploit the power of data analytics. For instance, we had to manually capture flight hour tracking data, which was time-consuming and prone to errors. If we could link this data directly to Power BI dashboards, it would significantly enhance operational efficiency within the aviation community. This integration could have enabled real-time pilot data management and more accurate tracking, but these opportunities were blocked by restrictive policies.

Creating a Data Management Role

It became clear that data integration can’t be treated as an additional duty. There is a growing need for the Army to establish formal data management roles within its units. Soldiers in these roles would need specialized training with a curriculum tailored to the military's unique data challenges. This would allow them to manage, analyze, and visualize data in a way that empowers leadership to make well-informed decisions.

Friction Points

One of the major friction points in data integration efforts within the brigade is restrictive cybersecurity protocols imposed by the RNEC. These restrictions prevent the installation of critical tools, hindering the ability to automate and streamline data analysis. This results in labor-intensive, manual processes that delay decision-making and reduce efficiency. Another challenge is the decentralized nature of data management across various sections. Different teams and leaders rely on their systems and methods for collecting and analyzing data, leading to a fragmented approach that creates inefficiencies and complicates the process of centralizing and standardizing data.

Solutions and Recommendations

To overcome some of these challenges, we should pursue a more flexible cybersecurity approach tailored to data analytics needs. Leadership should work with local RNECs to secure waivers for segregated environments, such as sandbox systems, where advanced tools can be used safely without compromising security.

A centralized data management framework needs to be implemented, including standardized data collection and reporting methods across all sections. Designating a brigade-level data officer to oversee this effort would ensure synchronization and data integrity. Offering training sessions for personnel to align with these new standards will further enhance data cohesion and operational effectiveness across the brigade.

A standardized training program should be implemented across the Army to equip leaders with the skills and capabilities to use these systems effectively. As the Army continues to advance in data-driven operations, leaders must be prepared with the knowledge to navigate current and future challenges in this evolving landscape.

Conclusion

Data integration within the Army is a complex but necessary evolution. Through our challenges, we have learned the importance of proper data education, centralized data management, and the need for dedicated personnel to manage these advanced systems.

Moving forward, we must prioritize flexibility in our IT infrastructure, streamline cybersecurity protocols, and provide comprehensive training to ensure that the Army fully capitalizes on the power of data analytics.