Release Notes for Data Sprint 20.11

Introduction

Date: 21 December 2020

The following list of new features, tasks, and resolved bugs are included in the latest Sprint 20.11.11 release to FDW/FDE:

Features

Design and implement import of WorldPop demographic data https://fewsnet.atlassian.net/browse/DATA-479
Create a PandasKMLRenderer https://fewsnet.atlassian.net/browse/DATA-546
Writable API for DataCollection, DataCollectionPeriod and DataPoint https://fewsnet.atlassian.net/browse/DATA-552
Managing weekly commodity availability data in the FDW https://fewsnet.atlassian.net/browse/DATA-568
Enhanced Country API for easier maintenance of visualizations https://fewsnet.atlassian.net/browse/DATA-569
Add verbiage on the login page indicating that we are currently not accepting new FDE account registrations https://fewsnet.atlassian.net/browse/DATA-599
Replicate the "Options" tab functionality that is currently in the Price domain (FDE) onto the Trade Domain in FDE https://fewsnet.atlassian.net/browse/DATA-600
Include a filter box for Flow Type on the Trade domain https://fewsnet.atlassian.net/browse/DATA-601
Create a DataCollectionFilter and DataCollectionPeriodFilter and update serializers https://fewsnet.atlassian.net/browse/DATA-624
IPC Map improvements https://fewsnet.atlassian.net/browse/DATA-625
IPC phase data validations https://fewsnet.atlassian.net/browse/DATA-627
Sending email notification to the user after Drupal publication https://fewsnet.atlassian.net/browse/DATA-630

Tasks

Reconciling crop production indictors of the crop_region and crop_unit for Afmadow, Lower Juba, Somalia Crop Reporting Unit https://fewsnet.atlassian.net/browse/DATA-453
Reload historic IPC data https://fewsnet.atlassian.net/browse/DATA-541
Reload all Pre-2018 IPC data into the system https://fewsnet.atlassian.net/browse/DATA-597

Bugs

Cross Border Trade data are not recognized from the data collection screen https://fewsnet.atlassian.net/browse/DATA-605
Semi-structured i.e (WFP VAM, Afghanistan: Days Work Available ) uploading issue https://fewsnet.atlassian.net/browse/DATA-614
Geographic filter of the FDE took longer minutes to load the data https://fewsnet.atlassian.net/browse/DATA-645
The IPCPackageViewSet doesn't support KMU documents correctly https://fewsnet.atlassian.net/browse/DATA-705
Market name not recognized in upload file https://fewsnet.atlassian.net/browse/DATA-709

Features

Design and implement import of WorldPop demographic data https://fewsnet.atlassian.net/browse/DATA-479

This feature is about loading the WorldPop data into Amazon’s S3 bucket and enhancing the APIs to get a population data extract in a tabular format (as seen here: https://fdw.fews.net/api/geographicunit/?format=csv&unit_type=fsc_admin_lhz&country_code=KE&as_of_date=2020-12-1&fields=with_population) and with a demographic breakdown (as seen here: https://fdw.fews.net/api/geographicunit/?format=csv&unit_type=fsc_admin_lhz&country_code=KE&as_of_date=2020-12-1&fields=with_population ).

Create a PandasKMLRenderer https://fewsnet.atlassian.net/browse/DATA-546

This enhancement is about allowing the KML format extract for the for DataPointViewSet, for instance, https://fdw.fews.net/api/cropproductionindicatorvalue/?format=kml&country_code=AF&cpcv2=R01.

Writable API for DataCollection, DataCollectionPeriod and DataPoint https://fewsnet.atlassian.net/browse/DATA-552

This feature is about enhancing the operations of creating, reading, updating and deleting of the existing data collection, data collection period, data point API end points from the back end.

Managing weekly commodity availability data in the FDW https://fewsnet.atlassian.net/browse/DATA-568

This feature is about allowing the import of the weekly commodity availability data into FDW and extract the data from the API end point of the marketpricefacts (i.e. https://fdw.fews.net/api/marketpricefacts/).

Enhanced Country API for easier maintenance of visualizations https://fewsnet.atlassian.net/browse/DATA-569

This feature is about enhancing the country API (https://fdw.fews.net/api/country/) to include all the key fields which help for easier maintenance of the visualization.

Add verbiage on the login page indicating that we are currently not accepting new FDE account registrations https://fewsnet.atlassian.net/browse/DATA-599

This feature is about introducing a verbiage on the login page indicating that we are currently not accepting new FDE account registrations (https://fdw.fews.net/accounts/signup/).

Replicate the "Options" tab functionality that is currently in the Price domain (FDE) onto the Trade Domain in FDE https://fewsnet.atlassian.net/browse/DATA-600

This enhancement is about adding the Options tab functionality to the Trade data domain of the FDE (https://fdw.fews.net/data-explorer/#/trade).

Include a filter box for Flow Type on the Trade domain https://fewsnet.atlassian.net/browse/DATA-601

This feature is about introducing Flow Type filter to the Trade data domain of the FDE (https://fdw.fews.net/data-explorer/#/trade).

Create a DataCollectionFilter and DataCollectionPeriodFilter and update serializers https://fewsnet.atlassian.net/browse/DATA-624

This feature is about adding the filters datasourcedocument_name,datasourceorganization_name , data_domain , datasourcedocument_source_type, country_code ,collection_status for the APIs (https://fdwdev.fews.net/api/datacollection/ and https://fdwdev.fews.net/api/datacollectionperiod/) which enables user to get specific data.

IPC Map improvements https://fewsnet.atlassian.net/browse/DATA-625

This enhancement is about improving the IPC map by supporting the remote monitoring countries and using an enhanced country API .

IPC phase data validations https://fewsnet.atlassian.net/browse/DATA-627

This feature is about introducing validations for IPC phases data imports.

Sending email notification to the user after Drupal publication https://fewsnet.atlassian.net/browse/DATA-630

This feature is about implementing email notification to the user who published the IPC classification data.

Tasks

Reconciling crop production indictors of the crop_region and crop_unit for Afmadow, Lower Juba, Somalia Crop Reporting Unit https://fewsnet.atlassian.net/browse/DATA-453

This task was about reconciling the crop production indicators i.e (crop_region and crop_unit) by changing the geographic unit types of Afmadow & Hagar to crop_region unit type .

Reload historic IPC data https://fewsnet.atlassian.net/browse/DATA-541

This task was about loading the historic FSO and FSOU data of the IPC Classification into FDW and the data was successfully loaded.

Reload all Pre-2018 IPC data into the system https://fewsnet.atlassian.net/browse/DATA-597

This task was about loading pre-2018 IPC historical data into the FDW Production environment (https://fdw.fews.net/en/admin/warehouse/datacollection/?data_domain=IPCClassification) and the data was loaded successfully.

Bugs

Cross Border Trade data are not recognized from the data collection screen https://fewsnet.atlassian.net/browse/DATA-605

There was an issue in loading cross border trade data from the data collection screen (https://fdw.fews.net/en/admin/warehouse/datacollection/add/) and this was resolved and the functionality is working as expected.

Semi-structured i.e (WFP VAM, Afghanistan: Days Work Available ) uploading issue https://fewsnet.atlassian.net/browse/DATA-614

There was a problem in loading the Days Work Available data and this was resolved and the final template for uploading the data has also been shared.

Geographic filter of the FDE took longer minutes to load the data https://fewsnet.atlassian.net/browse/DATA-645

There was an issue in loading the data values of the geographic filter of the crop data domain and this was resolved.

The IPCPackageViewSet doesn't support KMU documents correctly https://fewsnet.atlassian.net/browse/DATA-705

The downloaded KMU shapefiles were not showing the correct documents and this was resolved and it is working as expected .

Market name not recognized in upload file https://fewsnet.atlassian.net/browse/DATA-709

There was a problem in recognizing market in upload files and this was resolved and can now upload the files successfully.