Introduction to JRC CbM

• DIAS for CAP Checks by Monitoring
  • Main elements of the JRC CbM
  • DIAS and Copernicus Sentinel
  • CbM in the context of the CAP
  • JRC solutions for CbM
  • Contributing to JRC CbM
• System architecture
  • Scope of the CbM system
  • Cloud resources: DIAS infrastructure
    • What are DIAS
    • DIAS Virtual Machines
  • Sentinel data access: DIAS object storage
    • What is DIAS object storage
    • Copernicus ARD generation
  • Data repository - PostgreSQL/PostGIS
  • Extraction of statistics - Python processing
  • Data access - RESTful API
  • Interfaces for data use - Jupiter Notebooks
  • JRC CbM roles
• Backend
  • Goals
  • General requisites to set up the CbM system
  • How to create a CbM infrastructure
    • Links to the technical documentation
    • Set up a DIAS virtual machine
    • CbM database
    • CARD
    • Extract signatures
    • Create a RESTful API
  • Deployment of the CbM system
  • Expertise required
  • Enhancement Proposals
    • Terrain Correction
    • S2 Multi-band
    • Dask tests
    • ML Revision
    • RESTful meteo
• Frontend
  • Goals
  • Links to the technical documentation
  • RESTful API
  • Python CbM package
  • Expertise required
• Data analytics
  • Goals
  • Links to the technical documentation
  • Marker detection
  • Applications
  • Expertise required

Setup

• Setup DIAS VMs
  • Connecting to the ‘tenant host’ vm via SSH
• Software prerequisites
  • Docker (Ubuntu 20.04)
  • PostGIS
    • Optimizing
    • Essential CbM tables
  • Jupyter server
    • Build Jupyter image from source
• Database for JRC CbM
  • DB in the CbM architecture
  • Spatial database in a nutshell
    • Main database elements
    • The spatial bit
    • PostgreSQL and PostGIS
    • SQL
  • JRC CbM DB structure
  • Database access
    • Server/client structure
    • Access parameters
    • User access policy
  • Data retrieval
    • PgAdmin
    • phpPgAdmin
    • Psql
    • QGIS
    • Jupiter Notebook
    • RESTful API
    • R
  • Export and import data
  • Performance optimization
    • Basic optimization
    • Advanced optimization
• Data preparation
  • Adding shapefiles (.shp) to PostGIS database
  • Transfer metadata from the DIAS catalog
    • OpenSearch compliant catalogs (CREODIAS, MUNDI)
    • Database catalog (SOBLOO)
  • Benchmarking data formats for fast image processing
• Parcel extraction
  • Overview
  • Implementation details
  • Configuration files
  • Run with docker
  • Parallelization with docker swarm
    • Create VMs
    • Install docker and set up docker swarm
    • Configure and run
  • Caveats
• Backing up the database
  • Overview
  • Use pg_dump
• Build a RESTful API
  • Prerequisites
  • Create RESTful API users
  • Database connection
  • Dataset configuration
  • Deploy the RESTful API docker container
    • Build from source
  • Provide available options (Optional)
  • Adding orthophotos (Optional)

RESTful API

• Overview
• Parcel information
• Parcel Time Series
  • parcelTimeSeries
  • weatherTimeSeries
  • Simple python client to plot parcel Time Series
• Sentinel image chips
  • chipByLocation
  • rawChipByLocation
  • Example client code.
• Parcel orthophotos
• Parcel peers
  • parcelPeers
  • parcelStatsPeers
  • parcelsByPolygon
• Data analytics
  • parcel selection
  • time series statistics
  • histogram analysis, clustering
  • …
• POST requests
  • rawChipsBatch
  • Example test script
  • rawS1ChipsBatch

Use cases

• Calendar view
  • 1. Introduction
  • 2. Dependencies
    • 2.1 Calendar view Modules
  • 3 Structure of the code
  • 3.1 Initialization
  • 3.2 Main Processing Loop
  • 3.3 Output examples
• FOI Assessment
  • Concept
  • Methods and algorithms
  • Test results
  • Cardinality
  • FOI Clustering
• Machine learning
  • Introduction
  • Data preparation
    • Determine which classes to include
    • Composing the features for machine learning
    • Splitting in training and testing sets
  • Running the training and testing
  • Checking the results
    • Confusion matrix for testing sets
    • Combining different tensorflow runs
    • Identifying outliers
  • Next steps

Marker detection

• Overview of the marker detection tool
• Modules description
• Option file
• Graphical user interface
• Examples of marker detection

CbM python package

• Overview
  • Notebook widgets
  • Data stracture
  • Get widget
  • View widget
• Installation
  • Dependencies
  • Installing from PyPI
  • Installing for development
  • Installing from source:
  • Uninstallation
  • Install GDAL
  • Troubleshooting
• Configuration
  • Configuration widget
• DIAS catalog
  • Notebook widget
• Extraction
  • Extract widget
• FOI examples
  • Version 1
  • Version 2
• Data access
  • Parcel information
  • Time series
    • Show time series
    • Download time series
  • Background images
    • Show background images
    • Download background images
  • Sentinel chips

Developers

• Contributing to CbM
  • Using the issue tracker
  • Pull requests
  • Bug reports
  • License
• Pull requests