Release Notes for Data Sprint 21.08

Introduction

The following list of new features, tasks, and resolved bugs are included in the latest Sprint 21.08.4 release to FDW/FDE:

Features

Validate Geographic Unit for Calculated Price Index https://fewsnet.atlassian.net/browse/DATA-748
Prototype of clean PDFs from Wagtail pages https://fewsnet.atlassian.net/browse/DATA-945
DocumentTypeViewSet should only return document types with visible data https://fewsnet.atlassian.net/browse/DATA-986
Add an option to include Narrow Ranges in FIPE data domain extracts https://fewsnet.atlassian.net/browse/DATA-1033
Review Livelihood zones code column to use FNID instead of the LZ_CODE in uploads https://fewsnet.atlassian.net/browse/DATA-1035
Add admin 1 and admin 2 to the price time series extracts for both the price and CPI columns https://fewsnet.atlassian.net/browse/DATA-1047
Include Admin1 and admin2 in Market extracts in both the api end point and the admin extracts https://fewsnet.atlassian.net/browse/DATA-1057
Include datasource document in ExchangeRateResource and review all other dataseries resource classes to ensure they have source document in the import_fields list https://fewsnet.atlassian.net/browse/DATA-1059
Updated website link for Staple Food Price data download https://fewsnet.atlassian.net/browse/DATA-1063
Linear interpolation formulas for data points https://fewsnet.atlassian.net/browse/DATA-1065
Pull in Data from Colombia Price API https://fewsnet.atlassian.net/browse/DATA-1068
Market Price pipeline from KoboToolbox to FDW https://fewsnet.atlassian.net/browse/DATA-1074

Tasks

Unable to load NG_LHZ_2018 livelihood file https://fewsnet.atlassian.net/browse/DATA-446

Bugs

Market screen doesn't allow to make a change on a country name https://fewsnet.atlassian.net/browse/DATA-1058
Fix Resource Class to Return Lat/Long in Market Extracts https://fewsnet.atlassian.net/browse/DATA-1093
Error in "other" domain of FDE https://fewsnet.atlassian.net/browse/DATA-1095
FDE filter not working for labor "products" https://fewsnet.atlassian.net/browse/DATA-1098
Cross Border Trade (XBT) Transit recognition error https://fewsnet.atlassian.net/browse/DATA-1100

Features

Validate Geographic Unit for Calculated Price Index https://fewsnet.atlassian.net/browse/DATA-748

This enhancement is about adding a validation error message , i.e The {icm.name} Index Component is for {icm.geographic_unit} Geographic Unit which cannot be aggregated to the {dsr.geographic_unit} Geographic Unit for this Calculated Index , for when an Index Component is added to a Calculated Price Index, or if the Geographic Unit for a Calculated Price Index is changed.

Prototype of clean PDFs from Wagtail pages https://fewsnet.atlassian.net/browse/DATA-945

This feature is about developing a prototype for generating PDF reports which contains an interactive visualizations.

DocumentTypeViewSet should only return document types with visible data https://fewsnet.atlassian.net/browse/DATA-986

This enhancement is about showing the document types of the OTHER data domain (https://fdw.fews.net/data-explorer/#/other) which contains data for the external users.

Add an option to include Narrow Ranges in FIPE data domain extracts https://fewsnet.atlassian.net/browse/DATA-1033

The /api/ipcpopulationsize/ API end point has been enhanced to allow the range parameter which as a result accepts the point, narrow and public values; for instance ,

https://fdw.fews.net/api/ipcpopulationsize/?format=html&country_code=NG&range=point

https://fdw.fews.net/api/ipcpopulationsize/?format=html&country_code=NG&range=narrow and

https://fdw.fews.net/api/ipcpopulationsize/?format=html&country_code=NG&range=public

Review Livelihood zones code column to use FNID instead of the LZ_CODE in uploads https://fewsnet.atlassian.net/browse/DATA-1035

This feature is about enhancing the livelihood zones code column to use the FNID ; i.e (https://fdw.fews.net/en/admin/spatial/livelihoodzone/).

Add admin 1 and admin 2 to the price time series extracts for both the price and CPI columns https://fewsnet.atlassian.net/browse/DATA-1047

This enhancement is about adding admin 1 and admin 2 fields to the price timeseries extract , i.e (http://fdw.fews.net/api/marketpricefacts/?format=csv&timeseries=true&country_code=CM) and the metadata fields are revised based on the suggestions of the EWT i.e base_period_start - base_period_end is shown as a concatenated field and in addition to this , the period dates are shown in the newly added column;

Include Admin1 and admin2 in Market extracts in both the api end point and the admin extracts https://fewsnet.atlassian.net/browse/DATA-1057

This feature is about allowing the inclusions of Admin1 and Admin2 in the market API end point (https://fdw.fews.net/api/market/) and Admin (https://fdw.fews.net/en/admin/spatial/market/export/?) extracts.

Include datasource document in ExchangeRateResource and review all other dataseries resource classes to ensure they have source document in the import_fields list https://fewsnet.atlassian.net/browse/DATA-1059

This feature is about introducing datasourcedocument in all the import_fields of the data series resource classes , for instance , (https://fdw.fews.net/en/admin/price/exchangerate/import/).

Updated website link for Staple Food Price data download https://fewsnet.atlassian.net/browse/DATA-1063

This feature is about updating the website link by introducing additional parameters such as end_date=second_last_month or end_date=second_last_month, for instance ,(https://fdw.fews.net/api/marketpricefacts/?format=xlsx&country_code=ET&fields=website&end_date=last_month).

Linear interpolation formulas for data points https://fewsnet.atlassian.net/browse/DATA-1065

This enhancement is about implementing a linear interpolation formulas for a maximum of four consecutive missing data point values of a data series.

Pull in Data from Colombia Price API https://fewsnet.atlassian.net/browse/DATA-1068

This feature is about introducing the pipeline that imports daily data collections from the Colombia DANE API.

Market Price pipeline from KoboToolbox to FDW https://fewsnet.atlassian.net/browse/DATA-1074

This feature is about the ingestion of the price data collected using Kobo into FDW .The weekly data collected in kobo (forms.fews.net) and its status approved is automatically imported into FDW.

Tasks

Unable to load NG_LHZ_2018 livelihood file https://fewsnet.atlassian.net/browse/DATA-446

This task was about loading the NG_LHZ_2018 livelihood file and it was successfully loaded (https://fdw.fews.net/en/admin/spatial/geographicunitsetversion/3475/change/?_changelist_filters=geographicunitset__country__iso3166a2__exact%3DNG).

Bugs

Market screen doesn't allow to make a change on a country name https://fewsnet.atlassian.net/browse/DATA-1058

There was an issue in updating the country name of the market screen (https://fdw.fews.net/en/admin/spatial/market/) and this was resolved.

Fix Resource Class to Return Lat/Long in Market Extracts https://fewsnet.atlassian.net/browse/DATA-1093

The market extract was returning an empty columns for the metadata fields of Latitude and Longitude and this was fixed and can now access the values through the extract , i.e (https://fdw.fews.net/en/admin/spatial/market/).

Error in "other" domain of FDE https://fewsnet.atlassian.net/browse/DATA-1095

Trying to select data series from the OTHER data domain of the FDE results in a blank page and this was resolved and the functionality is working as expected.

FDE filter not working for labor "products" https://fewsnet.atlassian.net/browse/DATA-1098

There was a problem for filtering a specific products of Labor in the price data domain of the FDE, i.e (https://fdw.fews.net/data-explorer/#/price) and this was resolved.

Cross Border Trade (XBT) Transit recognition error https://fewsnet.atlassian.net/browse/DATA-1100

There was a problem in recognizing the Transit flow_type in the cross border trade data upload and this was resolved and the metadata successfully loaded.