Page Comparison

Introduction

Date: 11 August 2020

The following list of new features, tasks and resolved bugs are included in the Data Sprint 20.07 release which is running version 20.07.8:

Features

• Ingestion pipeline for Crop Production data - https://fewsnet.atlassian.net/browse/DATA-212

• Price Charts Dashboard Enhanced Version - https://fewsnet.atlassian.net/browse/DATA-273

• Add a "Best" option to the Scenario Filter for IPCPhaseViewSet - https://fewsnet.atlassian.net/browse/DATA-295

• Incorporate M&T Formatting Guidance into FDE Market Price and Cross-Border Trade Charts - https://fewsnet.atlassian.net/browse/DATA-306

• Enhance the Crop Production Dashboard to show Data Points by Product - https://fewsnet.atlassian.net/browse/DATA-324

• Population Estimates for ad hoc shapefiles - https://fewsnet.atlassian.net/browse/DATA-343

• Add Totals to the Cumulative Number of Data Points by Month - https://fewsnet.atlassian.net/browse/DATA-384

• Add as_of_date filter to DataPointViewSet - https://fewsnet.atlassian.net/browse/DATA-393

• Better Support to ADS-579 Compliance within FDW - https://fewsnet.atlassian.net/browse/DATA-402

• Introduce duplicate period date within the same Data Source Document for forecast domains - https://fewsnet.atlassian.net/browse/DATA-413

• Include data domain and dataseries count in DataSourceDocumentSerializer - https://fewsnet.atlassian.net/browse/DATA-438

• Apply pagination / Infinite Scroll to Semistructured data in Other domain - https://fewsnet.atlassian.net/browse/DATA-451

Tasks

• Complete Bulk Upload from the S3 Bucket Livelihood Zones into FDW - https://fewsnet.atlassian.net/browse/DATA-168

• In the map view while selecting a country, add a zoom-in feature so it takes you directly to that country, and the other countries around it are mostly hidden - https://fewsnet.atlassian.net/browse/DATA-426

• Layers need to be sorted alphabetically when viewing the spatial domain - https://fewsnet.atlassian.net/browse/DATA-427

• Add export format for .csv on the Export tab in the FDE Spatial view - https://fewsnet.atlassian.net/browse/DATA-428

• Add a preview url to the the documents_list in mayan - https://fewsnet.atlassian.net/browse/DATA-434

• Load remittances data into production - https://fewsnet.atlassian.net/browse/DATA-440

Bugs

The aliases are dropped off at the creation of the product codes - https://fewsnet.atlassian.net/browse/DATA-73

Crop data extract missing data area population and per capita - https://fewsnet.atlassian.net/browse/DATA-344

https://fdw.fews.net/en/admin/spatial/geographicunit/export/? results in 500 Internal server error - https://fewsnet.atlassian.net/browse/DATA-345

Price dataset extracts extracts produce wrong CPI values - https://fewsnet.atlassian.net/browse/DATA-424

Geography filter not displaying - https://fewsnet.atlassian.net/browse/DATA-435

Landscan Population failure with pre-2000 years - https://fewsnet.atlassian.net/browse/DATA-442

Update warehouse/templates/warehouse/datapointchart.html to use /api/datapoint instead of /api/datapointfacts - https://fewsnet.atlassian.net/browse/DATA-455

Features

Ingestion pipeline for Crop Production data. https://fewsnet.atlassian.net/browse/DATA-212

• This effort produced a prototype for implementing an ingestion pipeline for Crop Production data. With more than 4 million rows of data, this domain was a clear candidate to demonstrate the flexibility of FDW’s ingestion pipeline capabilities.

Price Charts Dashboard Enhanced Version. https://fewsnet.atlassian.net/browse/DATA-273

• This feature is about implementing an enhanced Price Charts dashboard to include the Price Watch and the Price Bulletin, and can be accessed at: https://fdw.fews.net/dashboard/price/.

...

Add a "Best" option to the Scenario Filter for IPCPhaseViewSet. https://fewsnet.atlassian.net/browse/DATA-295

• This feature is about adding the “Best Case options” scenario types i.e (Best Case in 3 months & Best Case in 9 months) to the Scenario filter in FDW.

Incorporate M&T Formatting Guidance into FDE Market Price and Cross-Border Trade Charts. https://fewsnet.atlassian.net/browse/DATA-306

• This enhancement is about implementing the Markets and Trade formatting standards in the Market Price and Cross-Border Trade data domains of the FDE charts.

Enhance the Crop Production Dashboard to show Data Points by Product. https://fewsnet.atlassian.net/browse/DATA-324

• This feature is about enhancing the Crop Production Dashboard (https://fdw.fews.net/dashboard/crop-production/) to show the crop data by product , Indicator, Admin level for all the countries of all the years. Each of these points are filterable as well.

...

Population Estimates for ad hoc shapefiles. https://fewsnet.atlassian.net/browse/DATA-343

• This feature is about generating population estimates per density area from the ad hoc shapefiles uploaded into the system.

...

Add Totals to the Cumulative Number of Data Points by Month. https://fewsnet.atlassian.net/browse/DATA-384

• This enhancement is about adding total number of data points to the Data Inventory Dashboard. This is now shown in Cumulative Number of Data Points section to the right of the graphic, and broken down by month.

...

Add as_of_date filter to DataPointViewSet. https://fewsnet.atlassian.net/browse/DATA-393

• This feature is about introducing an as_of_date filter to the DataPointViewSet in FDW which enables users to find the data points that are valid for a specific date (i.e. https://fdw.fews.net/api/marketpricefacts/?format=html&as_of_date=2018-01-01&dataseries=1795617&fields=simple).

Better Support to ADS-579 Compliance within FDW. https://fewsnet.atlassian.net/browse/DATA-402

• This feature is about providing justification and requirements where needed per the ADS-579 guidelines, regarding our data stored in FDW under the Data Usage Policy reason (https://fdw.fews.net/en/admin/warehouse/datausagepolicyreason/). Currently we have a list of available reasons in the system but until we get the final list of reasons/justifications from USAID, please use the entry: https://fdw.fews.net/en/admin/warehouse/datausagepolicyreason/7/change/ for a creating new FEWS NET Only data source documents.

Introduce duplicate period date within the same Data Source Document for forecast domains. https://fewsnet.atlassian.net/browse/DATA-413

• This feature is about allowing a duplicate period date within the same Data Source Document for the forecast data within the IPC data domains. This was previously prevented by the system, but is now allowable.

Include data domain and dataseries count in DataSourceDocumentSerializer. https://fewsnet.atlassian.net/browse/DATA-438

• This feature is about including document_type,  data_domain and dataseries_count fields in the DataSourceDocumentSerializer (i.e https://fdwdev.fews.net/api/datasourcedocument/).

Apply pagination / Infinite Scroll to Semistructured data in Other domain. https://fewsnet.atlassian.net/browse/DATA-451

• This feature is about introducing the infinite scroll or pagination functionality for the Other Data Domain.

...

Tasks

Complete Bulk Upload from the S3 Bucket Livelihood Zones into FDW. https://fewsnet.atlassian.net/browse/DATA-168

• This task was about giving support on uploading the Livelihood Zones data and the data was successfully loaded. However, the remaining data to upload is being addressed in singular tickets by country.

In the map view while selecting a country, add a zoom-in feature so it takes you directly to that country, and the other countries around it are mostly hidden. https://fewsnet.atlassian.net/browse/DATA-426

• This task was about zooming in to the country a user selects in Map view of the Spatial data domain, so that it’s clear what country the user is viewing (i.e. https://fdwdev.fews.net/data-explorer/#/spatial).

Layers need to be sorted alphabetically when viewing the Spatial domain. https://fewsnet.atlassian.net/browse/DATA-427

• This task was about sorting the Admin layers alphabetically on the map view of the Spatial data domain.

...

Add export format for .csv on the Export tab in the FDE Spatial view. https://fewsnet.atlassian.net/browse/DATA-428

• This task was about adding the .csv format to the Export tab of the FDE Spatial Data Domain.

...

Add a preview url to the the documents_list in mayan. https://fewsnet.atlassian.net/browse/DATA-434

• This task was about adding the document_preview url to the document list within mayan (i.e

  https://fdw.fews.net/documents/documentlist/ ) which helps to preview the documents or access them in a new browser window.

Load remittances data into production. https://fewsnet.atlassian.net/browse/DATA-440

• This task was about loading the remittance data into FDW via the data ingestion pipelines. It was successfully loaded and can also be extracted from the FDE end point once it is published.

Bugs

The aliases are dropped off at the creation of the product codes. https://fewsnet.atlassian.net/browse/DATA-73

• The aliases were dropped off at the creation of the CPC Product codes and now the functionality is working as expected.

Crop data extract missing data area population and per capita. https://fewsnet.atlassian.net/browse/DATA-344

• On the crop data extract, the column of information for population and value_per_capita were shown empty and this was resolved.

https://fdw.fews.net/en/admin/spatial/geographicunit/export/? results in 500 Internal server error. https://fewsnet.atlassian.net/browse/DATA-345

• The system threw a 500 Internal server error upon trying to export the Geographic units in .xlsx format (https://fdw.fews.net/en/admin/spatial/geographicunit/export/?). This was resolved and can now be exported in any formats.

Price dataset extracts extracts produce wrong CPI values. https://fewsnet.atlassian.net/browse/DATA-424

• There were incorrect CPI values that were shown in Price datasets extracts and this was resolved.

Geography filter not displaying. https://fewsnet.atlassian.net/browse/DATA-435

• There was an issue in loading the data values of the geographic filter of the crop data domain and this was resolved.

Landscan Population failure with pre-2000 years. https://fewsnet.atlassian.net/browse/DATA-442

• There was a problem in loading the crop extracts for the years prior to 2000 and this was resolved.

Update warehouse/templates/warehouse/datapointchart.html to use /api/datapoint instead of /api/datapointfacts. https://fewsnet.atlassian.net/browse/DATA-455

• There was an issue in viewing charts on the single data series page of FDW and this was resolved. The functionality is now working as expected.