Release Notes for Data Sprint 20.07

Introduction

Date: 11 August 2020

The following list of new features, tasks and resolved bugs are included in the Data Sprint 20.07 release which is running version 20.07.8:

Features

Ingestion pipeline for Crop Production data - https://fewsnet.atlassian.net/browse/DATA-212
Price Charts Dashboard Enhanced Version - https://fewsnet.atlassian.net/browse/DATA-273
Add a "Best" option to the Scenario Filter for IPCPhaseViewSet - https://fewsnet.atlassian.net/browse/DATA-295
Incorporate M&T Formatting Guidance into FDE Market Price and Cross-Border Trade Charts - https://fewsnet.atlassian.net/browse/DATA-306
Enhance the Crop Production Dashboard to show Data Points by Product - https://fewsnet.atlassian.net/browse/DATA-324
Population Estimates for ad hoc shapefiles - https://fewsnet.atlassian.net/browse/DATA-343
Add Totals to the Cumulative Number of Data Points by Month - https://fewsnet.atlassian.net/browse/DATA-384
Add as_of_date filter to DataPointViewSet - https://fewsnet.atlassian.net/browse/DATA-393
Better Support to ADS-579 Compliance within FDW - https://fewsnet.atlassian.net/browse/DATA-402
Introduce duplicate period date within the same Data Source Document for forecast domains - https://fewsnet.atlassian.net/browse/DATA-413
Include data domain and dataseries count in DataSourceDocumentSerializer - https://fewsnet.atlassian.net/browse/DATA-438
Apply pagination / Infinite Scroll to Semistructured data in Other domain - https://fewsnet.atlassian.net/browse/DATA-451

Tasks

Complete Bulk Upload from the S3 Bucket Livelihood Zones into FDW - https://fewsnet.atlassian.net/browse/DATA-168
In the map view while selecting a country, add a zoom-in feature so it takes you directly to that country, and the other countries around it are mostly hidden - https://fewsnet.atlassian.net/browse/DATA-426
Layers need to be sorted alphabetically when viewing the spatial domain - https://fewsnet.atlassian.net/browse/DATA-427
Add export format for .csv on the Export tab in the FDE Spatial view - https://fewsnet.atlassian.net/browse/DATA-428
Add a preview url to the the documents_list in mayan - https://fewsnet.atlassian.net/browse/DATA-434
Load remittances data into production - https://fewsnet.atlassian.net/browse/DATA-440

Bugs

The aliases are dropped off at the creation of the product codes - https://fewsnet.atlassian.net/browse/DATA-73

Crop data extract missing data area population and per capita - https://fewsnet.atlassian.net/browse/DATA-344

https://fdw.fews.net/en/admin/spatial/geographicunit/export/? results in 500 Internal server error - https://fewsnet.atlassian.net/browse/DATA-345

Price dataset extracts extracts produce wrong CPI values - https://fewsnet.atlassian.net/browse/DATA-424

Geography filter not displaying - https://fewsnet.atlassian.net/browse/DATA-435

Landscan Population failure with pre-2000 years - https://fewsnet.atlassian.net/browse/DATA-442

Update warehouse/templates/warehouse/datapointchart.html to use /api/datapoint instead of /api/datapointfacts - https://fewsnet.atlassian.net/browse/DATA-455

Features

Ingestion pipeline for Crop Production data. https://fewsnet.atlassian.net/browse/DATA-212

This effort produced a prototype for implementing an ingestion pipeline for Crop Production data. With more than 4 million rows of data, this domain was a clear candidate to demonstrate the flexibility of FDW’s ingestion pipeline capabilities.

Price Charts Dashboard Enhanced Version. https://fewsnet.atlassian.net/browse/DATA-273

This feature is about implementing an enhanced Price Charts dashboard to include the Price Watch and the Price Bulletin, and can be accessed at: https://fdw.fews.net/dashboard/price/.

Add a "Best" option to the Scenario Filter for IPCPhaseViewSet. https://fewsnet.atlassian.net/browse/DATA-295

This feature is about adding the “Best Case options” scenario types i.e (Best Case in 3 months & Best Case in 9 months) to the Scenario filter in FDW.

Incorporate M&T Formatting Guidance into FDE Market Price and Cross-Border Trade Charts. https://fewsnet.atlassian.net/browse/DATA-306

This enhancement is about implementing the Markets and Trade formatting standards in the Market Price and Cross-Border Trade data domains of the FDE charts.

Enhance the Crop Production Dashboard to show Data Points by Product. https://fewsnet.atlassian.net/browse/DATA-324

This feature is about enhancing the Crop Production Dashboard (https://fdw.fews.net/dashboard/crop-production/) to show the crop data by product , Indicator, Admin level for all the countries of all the years. Each of these points are filterable as well.

Population Estimates for ad hoc shapefiles. https://fewsnet.atlassian.net/browse/DATA-343

This feature is about generating population estimates per density area from the ad hoc shapefiles uploaded into the system.

Add Totals to the Cumulative Number of Data Points by Month. https://fewsnet.atlassian.net/browse/DATA-384

This enhancement is about adding total number of data points to the Data Inventory Dashboard. This is now shown in Cumulative Number of Data Points section to the right of the graphic, and broken down by month.

Add as_of_date filter to DataPointViewSet. https://fewsnet.atlassian.net/browse/DATA-393

This feature is about introducing an as_of_date filter to the DataPointViewSet in FDW which enables users to find the data points that are valid for a specific date (i.e. https://fdw.fews.net/api/marketpricefacts/?format=html&as_of_date=2018-01-01&dataseries=1795617&fields=simple).

Better Support to ADS-579 Compliance within FDW. https://fewsnet.atlassian.net/browse/DATA-402

This feature is about providing justification and requirements where needed per the ADS-579 guidelines, regarding our data stored in FDW under the Data Usage Policy reason (https://fdw.fews.net/en/admin/warehouse/datausagepolicyreason/). Currently we have a list of available reasons in the system but until we get the final list of reasons/justifications from USAID, please use the entry: https://fdw.fews.net/en/admin/warehouse/datausagepolicyreason/7/change/ for a creating new FEWS NET Only data source documents.

Introduce duplicate period date within the same Data Source Document for forecast domains. https://fewsnet.atlassian.net/browse/DATA-413

This feature is about allowing a duplicate period date within the same Data Source Document for the forecast data within the IPC data domains. This was previously prevented by the system, but is now allowable.

Include data domain and dataseries count in DataSourceDocumentSerializer. https://fewsnet.atlassian.net/browse/DATA-438

This feature is about including document_type, data_domain and dataseries_count fields in the DataSourceDocumentSerializer (i.e https://fdwdev.fews.net/api/datasourcedocument/).

Apply pagination / Infinite Scroll to Semistructured data in Other domain. https://fewsnet.atlassian.net/browse/DATA-451

This feature is about introducing the infinite scroll or pagination functionality for the Other Data Domain.

Tasks

Complete Bulk Upload from the S3 Bucket Livelihood Zones into FDW. https://fewsnet.atlassian.net/browse/DATA-168

This task was about giving support on uploading the Livelihood Zones data and the data was successfully loaded. However, the remaining data to upload is being addressed in singular tickets by country.

In the map view while selecting a country, add a zoom-in feature so it takes you directly to that country, and the other countries around it are mostly hidden. https://fewsnet.atlassian.net/browse/DATA-426

This task was about zooming in to the country a user selects in Map view of the Spatial data domain, so that it’s clear what country the user is viewing (i.e. https://fdwdev.fews.net/data-explorer/#/spatial).

Layers need to be sorted alphabetically when viewing the Spatial domain. https://fewsnet.atlassian.net/browse/DATA-427

This task was about sorting the Admin layers alphabetically on the map view of the Spatial data domain.

Add export format for .csv on the Export tab in the FDE Spatial view. https://fewsnet.atlassian.net/browse/DATA-428

This task was about adding the .csv format to the Export tab of the FDE Spatial Data Domain.

Add a preview url to the the documents_list in mayan. https://fewsnet.atlassian.net/browse/DATA-434

This task was about adding the document_preview url to the document list within mayan (i.e
https://fdw.fews.net/documents/documentlist/ ) which helps to preview the documents or access them in a new browser window.

Load remittances data into production. https://fewsnet.atlassian.net/browse/DATA-440

This task was about loading the remittance data into FDW via the data ingestion pipelines. It was successfully loaded and can also be extracted from the FDE end point once it is published.

Bugs

The aliases are dropped off at the creation of the product codes. https://fewsnet.atlassian.net/browse/DATA-73

The aliases were dropped off at the creation of the CPC Product codes and now the functionality is working as expected.

Crop data extract missing data area population and per capita. https://fewsnet.atlassian.net/browse/DATA-344

On the crop data extract, the column of information for population and value_per_capita were shown empty and this was resolved.

https://fdw.fews.net/en/admin/spatial/geographicunit/export/? results in 500 Internal server error. https://fewsnet.atlassian.net/browse/DATA-345

The system threw a 500 Internal server error upon trying to export the Geographic units in .xlsx format (https://fdw.fews.net/en/admin/spatial/geographicunit/export/?). This was resolved and can now be exported in any formats.

Price dataset extracts extracts produce wrong CPI values. https://fewsnet.atlassian.net/browse/DATA-424

There were incorrect CPI values that were shown in Price datasets extracts and this was resolved.

Geography filter not displaying. https://fewsnet.atlassian.net/browse/DATA-435

There was an issue in loading the data values of the geographic filter of the crop data domain and this was resolved.

Landscan Population failure with pre-2000 years. https://fewsnet.atlassian.net/browse/DATA-442

There was a problem in loading the crop extracts for the years prior to 2000 and this was resolved.

Update warehouse/templates/warehouse/datapointchart.html to use /api/datapoint instead of /api/datapointfacts. https://fewsnet.atlassian.net/browse/DATA-455

There was an issue in viewing charts on the single data series page of FDW and this was resolved. The functionality is now working as expected.