Background
The primary objective of the Force Availability and Inventory Data Curation Support Service is to
organize, clean, and standardize NATO military force and availability data to enhance its quality and usability for military exploitation.
The service will facilitate a campaign of recurring data curation sprints in support of NCIA
Operational Analysis personnel.These data curation sprints will establish accessible, consistent and reliable forces and inventory datasets for exploitation by existing NATO Operations and Defence Planning software tools. These datasets will be exploited for a range of military-analytic purposes.Goal is to ensure that raw Force Availability and Inventory data reported by nations to NATO is cleaned and processed efficiently.This will maximise accuracy, completeness, and standardisation of format.
The Data Curation Support Service should deliver innovative and tailored techniques and tools
emerging from on-going Artificial Intelligence/Big Data revolution.Intent is to support NCIA Operational Analysis to work smarter, not harder, as a force multiplier to significantly enhance the quality of NATO military forces and availability data.
The Data Curation Support Service solution will tailor established data curation techniques to clean,
transform, and validate data declared to NATO by the nations.The Data Curation Support Service will select the appropriate tools and techniques for data curation. The proposed service solution should offer robust capabilities for cleaning and standardizing data, along with facilitating complex data transformations. Data validation is a critical process in data curation that ensures the accuracy, consistency, and reliability of data. It involves checking data against predefined rules, constraints, or formats to identify and rectify errors. The Data Curation Support Service must apply validation techniques such as data profiling, schema validation, and rule-based validation to ensure data quality, and deliver integration tools to streamline the process of combining data from multiple sources. By leveraging these tools and techniques, the Data Curation Support Service will significantly enhance the quality and usability of the NATO forces and inventories datasets, supporting better decision-making and analytical outcomes.
Duties/Role
- Document the data curation process, create data dictionaries, metadata documentation, and final reports
- Effort Estimate: 12 person-days (1 person, part-time with travel to NCIA The Hague (NLD))
- Duration: 3 Weeks (15 working days) for sprint execution. Additional up-front preparation time
Deliverables
- A cleaned and standardized Forces & Inventory dataset ready for analysis and exploitation
- Documentation detailing the data curation process, methodologies used, and decisions made
- A report highlighting key findings and any issues encountered during the curation process
Outline Plan
- Week 1: Planning and Initial Assessment
- Day 1: FORCES & INVENTORY Data Curation Sprint Kick-off
ü Introduce team members and their roles
ü Outline Forces & Inventory Data Curation Sprint objectives, scope, and deliverables
ü Establish communication channels and tools for collaboration
- Day 2-3: Data Assessment and Inventory
ü Collect and inventorise the dataset to be curated
ü Perform an initial assessment to identify data quality issues such as missing values, duplicates, and inconsistencies
ü Document the findings and develop a preliminary plan for data curation
- Day 4-5: Strategy Development
ü Develop a detailed data curation approach based on the initial assessment
ü Define data quality criteria and standards to be achieved
ü Select tools and techniques for data cleaning, transformation, and validation
- Week 2: Data Cleaning and Transformation
- Day 6-8: Data Cleaning
ü Begin the data cleaning process by addressing the most critical issues identified in the assessment phase
ü Remove duplicates, correct errors, and handle missing values
ü Ensure consistency in data formats and values
- Day 9-10: Data Transformation
ü Standardize data formats, such as date and time formats, numerical values, and categorical variables
ü Normalize data where necessary to ensure consistency across the dataset
ü Apply any necessary data enrichment processes, such as adding derived variables or external data
- Day 10: Mid-Sprint Review
ü Conduct a mid-sprint review to assess progress
ü Discuss any challenges encountered and adjust the strategy as needed
ü Ensure alignment with Forces & Inventory Data Curation Sprint objectives and timelines
- Week 3: Validation, Documentation, and Reporting
- Day 11-13: Data Validation
ü Validate the curated data against the defined quality criteria
ü Perform checks for accuracy, completeness, and consistency
ü Conduct peer reviews to ensure the data meets the required standards
ü Document the data curation process, including methodologies used, issues encountered, and resolutions implemented
ü Create a data dictionary and metadata documentation for the curated dataset
- Day 15: Final Review and Reporting
ü Conduct a final review of the curated dataset
ü Prepare a comprehensive report summarizing the data curation process, key findings, and any recommendations for future data management
ü Present the curated dataset and documentation to stakeholders for feedback and approval