Release Notes for Data Sprint 21.06

Introduction

The following list of new features, tasks, and resolved bugs are included in the latest Sprint 21.06 release to FDW/FDE:

Features

• Investigate dissolve performance in /api/ipcpasemap - https://fewsnet.atlassian.net/browse/DATA-621

• Replace Staple Food Price Data Set on Data center with direct download from FDW - https://fewsnet.atlassian.net/browse/DATA-906

• Review the pre-processing and matching of DataSourceDocuments in uploads - https://fewsnet.atlassian.net/browse/DATA-935

• Add collection_start_date and collection_end_date filters to DataPointFilter - https://fewsnet.atlassian.net/browse/DATA-968

• Support fields parameter for geojson extracts of DataPointViewSet - https://fewsnet.atlassian.net/browse/DATA-969

• Support As-Is imports on the three usual import screens https://fewsnet.atlassian.net/browse/DATA-971

• As a Food Security Analyst, I want to be able to view the admin 1 & 2 names when I extract market price data https://fewsnet.atlassian.net/browse/DATA-974

•  As a Food Security Analyst, I want to be able to view the CPI subnational data associated with an admin 1 in my extract https://fewsnet.atlassian.net/browse/DATA-975

Tasks

• Trade Flow resource generates lots of duplicate queries https://fewsnet.atlassian.net/browse/DATA-950

• Enable extraction of reusable visualizations JSON format - https://fewsnet.atlassian.net/browse/DATA-993

Bugs

• Zimbabwe common currency prices not being calculated in data extracts - https://fewsnet.atlassian.net/browse/DATA-533

• Fix price charts located under the “Old Charts” tab - https://fewsnet.atlassian.net/browse/DATA-560

• Error uploading ET 2018 and 2009 LHZ shapefiles - https://fewsnet.atlassian.net/browse/DATA-784

• Investigate potential templates (auto-complete field) loading issue with the new Autocomplete functionality - https://fewsnet.atlassian.net/browse/DATA-957

• MarketProduct search doesn’t recognize partial Price Types - https://fewsnet.atlassian.net/browse/DATA-961

• Error extracting Haiti and Madagascar availability data - https://fewsnet.atlassian.net/browse/DATA-1000

• Collection date field is not automatically populated on the data collection page - https://fewsnet.atlassian.net/browse/DATA-1002

Features

Investigate dissolve performance in /api/ipcpasemap - https://fewsnet.atlassian.net/browse/DATA-621

• This feature improved the performance of GeoJSON downloads for map visualizations by dissolving internal boundaries prior to extraction so that the download doesn't need to contain any of the "internal" boundaries under each phase; for instance ,(https://fdw.fews.net/api/ipcphasemap/?format=geojson&country_code=MZ&collection_date=2020-10-01&scenario=ML1&_refresh=true , https://fdw.fews.net/api/ipcphasemap/?format=geojson&country_code=NG&collection_date=2020-10-01&scenario=ML1&_refresh=true and https://fdw.fews.net/api/ipcphasemap/?format=geojson&country_code=MG&collection_date=2020-10-01&scenario=ML1&_refresh=true).

Replace Staple Food Price Data Set on Data center with direct download from FDW - https://fewsnet.atlassian.net/browse/DATA-906

• This implementation allows the direct download of the price data file (https://fdw.fews.net/api/marketprice.xlsx/?dataset=FEWS_NET_Price_Data )

  and the downloaded file should also contain the selected fields fit for the website extracts and a read me tab as well.

Review the pre-proessing and matching of DataSourceDocuments in uploads - https://fewsnet.atlassian.net/browse/DATA-935

• This feature is about making the data_source_document with a trailing space to be recognized in an uploading process.

Add collection_start_date and collection_end_date flters to DataPointFilter - https://fewsnet.atlassian.net/browse/DATA-968

• This feature enables users to easily update local data visualizations each month via the API by using the collection_start_date and collection_end_date filters in the Data Point Filter; for instance ,(https://fdw.fews.net/api/ipcphase/?format=geojson&collection_date_start=2021-04-01&country_code=KE , https://fdw.fews.net/api/datacollectionperiod/?format=json&collection_start_date=2021-04-01&data_domain=IPCCLASSIFICATION and https://fdw.fews.net/api/datacollection/?format=json&collection_start_date=2021-04-01&data_domain=IPCCLASSIFICATION )

Support fields parameter for geojson extracts of DataPointViewSet - https://fewsnet.atlassian.net/browse/DATA-969

• This enhancement is about incorporating fields parameter for GeoJSON extracts of the DataPointViewSet API endpoint , for instance; (https://fdw.fews.net/api/ipcphase/?format=geojson&collection_date=2020-10-01&country_code=KE&fields=id,scenario,start_date,end_date,collection_date,value)

Support As-Is imports on the three usual import screens https://fewsnet.atlassian.net/browse/DATA-971

• This feature is about implementing the As-Is imports in three import screens ;

1. Offline Data Uploads screen (https://fdw.fews.net/en/admin/warehouse/dataupload/add/)

2. Add Data Collection screen  (https://fdw.fews.net/en/admin/warehouse/datacollection/add/)

3. Semi-Structured Data Point Admin(https://fdw.fews.net/en/admin/semistructured/asisdatapoint/import/)

As a Food Security Analyst, I want to be able to view the admin 1 & 2 names when I extract market price data https://fewsnet.atlassian.net/browse/DATA-974

• This feature is about allowing the inclusions of the names of the Admin1 and Admin2 in the market price data extract .

As a Food Security Analyst, I want to be able to view the CPI subnational data associated with an admin 1 in my extract https://fewsnet.atlassian.net/browse/DATA-975

• This enhancement is about showing the data of the CPI subnational which are linked with Admin1 in the data that will be extracted.

Tasks

Trade Flow resource generates lots of duplicate queries https://fewsnet.atlassian.net/browse/DATA-950

• Since the trade flow resource generates multiple queries which will make the data upload process slow so this task was about generating fewer queries which makes the data upload process faster as a result .

Enable extraction of reusable visualizations JSON format - https://fewsnet.atlassian.net/browse/DATA-993

• This task was about allowing the extraction of a reusable visualization in a JSON format , for instance , i.e

  https://fdw.fews.net/viz/viz_visualizations_price_augmentedpricechart.json?dataseries=314489 https://fdw.fews.net/viz/viz_visualizations_price_currentprevioushistoricpricechart.json?dataseries__in=314459&historic_range=2y https://fdw.fews.net/viz/viz_visualizations_price_averagepricechangemap.json?dataseries__marketproduct__product=R01112AA&historic_range=1m&dataseries__geographic_unit__country=SD&period_date=2021-02-28

Bugs

Zimbabwe common currency prices not being calculated in data extracts - https://fewsnet.atlassian.net/browse/DATA-533

• The Zimbabwe common currency prices were not being calculated or presented in data extracts. This issue was resolved.

Fix price charts located under the “Old Charts” tab - https://fewsnet.atlassian.net/browse/DATA-560

• There was an issue in augmented price chart and this was resolved. The x-axis label has been reviewed, three tab names have also been changed.

Error uploading ET 2018 and 2009 LHZ shapefiles - https://fewsnet.atlassian.net/browse/DATA-784

• Users received an internal server error when attempting to upload ET 2018 and 2009 LHZ shapefiles. This issue was resolved.

Investigate potential templates (auto-complete field) loading issue with the new Autocomplete functionality - https://fewsnet.atlassian.net/browse/DATA-957

• There was a problem in selecting data values for some pages of the FDW, for instance , (https://fdw.fews.net/en/admin/warehouse/datacollection/add/ ,

  https://fdw.fews.net/en/admin/price/marketproduct/add/ ,

  https://fdw.fews.net/en/admin/price/exchangerate/add/ ,

  https://fdw.fews.net/en/admin/price/priceindex/add/ and https://fdw.fews.net/en/admin/warehouse/datasourcedocument/add/) due to the implementation of the new autocomplete functionality and this was resolved and all the pages are working as expected.

MarketProduct search doesn’t recognize partial Price Types - https://fewsnet.atlassian.net/browse/DATA-961

• There was a problem in  /api/marketproduct/ end point  in making search with a partial terms of the price type , for instance with whol instead of the full one wholesale or ret instead of the full retail and this was resolved and can be searching using , (https://fdw.fews.net/api/marketproduct/?format=json&search=eth mai ret
  https://fdw.fews.net/api/marketproduct/?format=json&search=nair mai whol ) .

Error extracting Haiti and Madagascar availability data - https://fewsnet.atlassian.net/browse/DATA-1000

• There was a problem in extracting the availability data of the countries Haiti and Madagascar  and this was resolved.

Collection date field is not automatically populated on the data collection page - https://fewsnet.atlassian.net/browse/DATA-1002

• The collection_date field was not populated automatically while uploading data on the data collection entry page (https://fdw.fews.net/en/admin/warehouse/datacollection/add/). This was resolved and the functionality is working as  expected.