Release Notes for Data Sprint 20.07

Introduction

The following list of new features, tasks and resolved bugs are included in the Data Sprint 20.07 release:

Features

• Ingestion pipeline for Crop Production datahttps://fewsnet.atlassian.net/browse/DATA-212

• Price Charts Dashboard Enhanced Version https://fewsnet.atlassian.net/browse/DATA-273

• Add a "Best" option to the Scenario Filter for IPCPhaseViewSet https://fewsnet.atlassian.net/browse/DATA-295

• Incorporate M&T Formatting Guidance into FDE Market Price and Cross-Border Trade Charts https://fewsnet.atlassian.net/browse/DATA-306

• Enhance the Crop Production Dashboard to show Data Points by Product https://fewsnet.atlassian.net/browse/DATA-324

• Population Estimates for ad hoc shapefiles https://fewsnet.atlassian.net/browse/DATA-343

• Add Totals to the Cumulative Number of Data Points by Month https://fewsnet.atlassian.net/browse/DATA-384

• Add as_of_date filter to DataPointViewSet https://fewsnet.atlassian.net/browse/DATA-393

• Better Support to ADS-579 Compliance within FDW https://fewsnet.atlassian.net/browse/DATA-402

• Introduce duplicate period date within the same Data Source Document for forecast domains https://fewsnet.atlassian.net/browse/DATA-413

• Include data domain and dataseries count in DataSourceDocumentSerializer https://fewsnet.atlassian.net/browse/DATA-438

• Apply pagination / Infinite Scroll to Semistructured data in Other domain https://fewsnet.atlassian.net/browse/DATA-451

Tasks

• Complete Bulk Upload from the S3 Bucket Livelihood Zones into FDW https://fewsnet.atlassian.net/browse/DATA-168

• In the map view while selecting a country, add a zoom-in feature so it takes you directly to that country, and the other countries around it are mostly hidden https://fewsnet.atlassian.net/browse/DATA-426

• Layers need to be sorted alphabetically when viewing the spatial domain https://fewsnet.atlassian.net/browse/DATA-427

• Add export format for .csv on the Export tab in the FDE Spatial view https://fewsnet.atlassian.net/browse/DATA-428

• Add a preview url to the the documents_list in mayan https://fewsnet.atlassian.net/browse/DATA-434

• Load remittances data into production https://fewsnet.atlassian.net/browse/DATA-440

Bugs

The aliases are dropped off at the creation of the product codes https://fewsnet.atlassian.net/browse/DATA-73

Crop data extract missing data area population and per capita https://fewsnet.atlassian.net/browse/DATA-344

https://fdw.fews.net/en/admin/spatial/geographicunit/export/? results in 500 Internal server error https://fewsnet.atlassian.net/browse/DATA-345

Price dataset extracts extracts produce wrong CPI values https://fewsnet.atlassian.net/browse/DATA-424

Geography filter not displaying https://fewsnet.atlassian.net/browse/DATA-435

Landscan Population failure with pre-2000 years https://fewsnet.atlassian.net/browse/DATA-442

Update warehouse/templates/warehouse/datapointchart.html to use /api/datapoint instead of /api/datapointfacts https://fewsnet.atlassian.net/browse/DATA-455

Features

Ingestion pipeline for Crop Production datahttps://fewsnet.atlassian.net/browse/DATA-212

• This feature is a prototype for implementing ingestion pipeline for Crop Production data.

Price Charts Dashboard Enhanced Version https://fewsnet.atlassian.net/browse/DATA-273

• This feature is about implementing an enhanced Price charts Dashboard such as the Price watch and the Price bulletin ( https://fdw.fews.net/dashboard/price/).

Add a "Best" option to the Scenario Filter for IPCPhaseViewSet https://fewsnet.atlassian.net/browse/DATA-295

• This feature is about adding the Best case options scenario types i.e (Best Case in 3 months & Best Case in 9 months) to Scenario filter.

Incorporate M&T Formatting Guidance into FDE Market Price and Cross-Border Trade Charts https://fewsnet.atlassian.net/browse/DATA-306

• This enhancement is about implementing the M&T formatting Guidance in the Market Price and Cross-Border Trade data domains of the FDE charts.

Enhance the Crop Production Dashboard to show Data Points by Product https://fewsnet.atlassian.net/browse/DATA-324

• This feature is about enhancing the Crop Production Dashboard (https://fdw.fews.net/dashboard/crop-production/ ) to show the crop data by product , Indicator, Admin level for all the countries of all the years.

Population Estimates for ad hoc shapefiles https://fewsnet.atlassian.net/browse/DATA-343

• This feature is about generating  population estimates per density area from the ad hoc shapefiles uploaded into the system.

Add Totals to the Cumulative Number of Data Points by Month https://fewsnet.atlassian.net/browse/DATA-384

• This enhancement is about adding Total number of data points to the Cumulative Number of Data Points by Month.

Add as_of_date filter to DataPointViewSet https://fewsnet.atlassian.net/browse/DATA-393

• This feature is about introducing an as_of_date filter to DataPointViewSet which enables to find the data points valid for a specific date, for instance ,(https://fdw.fews.net/api/marketpricefacts/?format=html&as_of_date=2018-01-01&dataseries=1795617&fields=simple).

Better Support to ADS-579 Compliance within FDW https://fewsnet.atlassian.net/browse/DATA-402

• This feature allows to store data under Usage policy reason (https://fdw.fews.net/en/admin/warehouse/datausagepolicyreason/) and Usage policy justification for a data source document with a FEWS NET Only data source document type. And currently we have some list of available reasons in the system but until we get the final list of reasons from USAID , please use the entry (https://fdw.fews.net/en/admin/warehouse/datausagepolicyreason/7/change/ ) for a newly created FEWS NET Only data source documents.

Introduce duplicate period date within the same Data Source Document for forecast domains https://fewsnet.atlassian.net/browse/DATA-413

• This feature is about allowing duplicate period date with in the same Data source document for the forecast data domains.

Include data domain and dataseries count in DataSourceDocumentSerializer https://fewsnet.atlassian.net/browse/DATA-438

• This feature is about including document_type,  data_domain and dataseries_count fields in the DataSourceDocumentSerializer i.e (https://fdwdev.fews.net/api/datasourcedocument/).

Apply pagination / Infinite Scroll to Semistructured data in Other domain https://fewsnet.atlassian.net/browse/DATA-451

• This feature is about introducing the infinite scroll or pagination functionality for the OTHER data domain.

Tasks

Complete Bulk Upload from the S3 Bucket Livelihood Zones into FDW https://fewsnet.atlassian.net/browse/DATA-168

• This task was about giving support on uploading the Livelihood Zones data and the data was successfully loaded.

In the map view while selecting a country, add a zoom-in feature so it takes you directly to that country, and the other countries around it are mostly hidden https://fewsnet.atlassian.net/browse/DATA-426

• This task was about introducing the zoom-in feature to the Map view of the Spatial data domain (https://fdwdev.fews.net/data-explorer/#/spatial).

Layers need to be sorted alphabetically when viewing the spatial domain https://fewsnet.atlassian.net/browse/DATA-427

• This task was about sorting the Admin layers alphabetically on the map view of the spatial data domain.

Add export format for .csv on the Export tab in the FDE Spatial view https://fewsnet.atlassian.net/browse/DATA-428

• This task was about adding the .csv format on the Export tab of the FDE Spatial Data Domain.

Add a preview url to the the documents_list in mayan https://fewsnet.atlassian.net/browse/DATA-434

• This task was about adding document_preview url to the document list of mayan i.e

  (https://fdw.fews.net/documents/documentlist/ ) which helps to preview or access the documents in a new browser.

Load remittances data into production https://fewsnet.atlassian.net/browse/DATA-440

• This task was about loading the remittance data and it was successfully loaded & can also be extracted from the FDE end point.

Bugs

The aliases are dropped off at the creation of the product codes https://fewsnet.atlassian.net/browse/DATA-73

• The aliases were dropped off at the creation of the CPC Product codes and this was resolved & the functionality is working as expected.

Crop data extract missing data area population and per capita https://fewsnet.atlassian.net/browse/DATA-344

• On the crop data extract , the column information of population and value_per_capita columns were shown empty and this was resolved.

https://fdw.fews.net/en/admin/spatial/geographicunit/export/? results in 500 Internal server error https://fewsnet.atlassian.net/browse/DATA-345

• It was the 500 Internal server error which were shown upon trying to export the Geographic units in .xlsx format (https://fdw.fews.net/en/admin/spatial/geographicunit/export/?) and this was resolved & can now be exported in any formats.

Price dataset extracts extracts produce wrong CPI values https://fewsnet.atlassian.net/browse/DATA-424

• There were wrong CPI values which were shown in price datasets extracts and this was resolved.

Geography filter not displaying https://fewsnet.atlassian.net/browse/DATA-435

• There was an issue in loading the data values of the Geographic filter of the crop data domain and this was resolved.

Landscan Population failure with pre-2000 years https://fewsnet.atlassian.net/browse/DATA-442

• There was a problem in loading the crop extracts for the pre-2000 years and this was resolved.

Update warehouse/templates/warehouse/datapointchart.html to use /api/datapoint instead of /api/datapointfacts https://fewsnet.atlassian.net/browse/DATA-455

• There was an issue in viewing charts on the single data series page of the FDW and this was resolved & the functionality is working as expected.