Page Comparison

Dated: 12 July 2021

Introduction

The following list of new features, tasks, and resolved bugs are included in the latest Sprint 21.06.1 release to FDW/FDE:

Features

• Review the pre-processing and matching of DataSourceDocuments in uploads - https://fewsnet.atlassian.net/browse/DATA-935

• Add collection_start_date and collection_end_date filters to DataPointFilter - Investigate dissolve performance in /api/ipcpasemap - https://fewsnet.atlassian.net/browse/DATA-968621

• Replace Staple Food Price Data Set on Data center with direct download from FDW - https://fewsnet.atlassian.net/browse/DATA-906

• Investigate dissolve performance in /api/ipcpasemap Review the pre-processing and matching of DataSourceDocuments in uploads - https://fewsnet.atlassian.net/browse/DATA-621Standard Price Charts in Superset 935

• Add collection_start_date and collection_end_date filters to DataPointFilter - https://fewsnet.atlassian.net/browse/DATA-987968

• Support fields parameter for geojson extracts of DataPointViewSet - https://fewsnet.atlassian.net/browse/DATA-969 Add Component Tests for FDE2 -

• Support As-Is imports on the three usual import screens https://fewsnet.atlassian.net/browse/DATA-960Managing ZW exchange rate data in the FDW - 971

• As a Food Security Analyst, I want to be able to view the admin 1 & 2 names when I extract market price data https://fewsnet.atlassian.net/browse/DATA-617

Tasks

• Enable extraction of reusable visualizations JSON format - 974

•  As a Food Security Analyst, I want to be able to view the CPI subnational data associated with an admin 1 in my extract https://fewsnet.atlassian.net/browse/DATA-993Upgrade Django to 3.2 - 975

Tasks

• Trade Flow resource generates lots of duplicate queries https://fewsnet.atlassian.net/browse/DATA-942Confirm IPC Node checking notification works in PROD 950

• Enable extraction of reusable visualizations JSON format - https://fewsnet.atlassian.net/browse/DATA-988Support to Sudan Farmers API Testing 993

Bugs

• Zimbabwe common currency prices not being calculated in data extracts - https://fewsnet.atlassian.net/browse/DATA-897Investigation: Plotly cs mapbox vs Echart for the FDE2 map component 533

• Fix price charts located under the “Old Charts” tab - https://fewsnet.atlassian.net/browse/DATA-984

Bugs

• Semi structured domain api/Deomographyvalue endpoint CSV format raises KeyError - https://fewsnet.atlassian.net/browse/DATA-848 560

• Error uploading ET 2018 abd and 2009 LHZ shapefiles - https://fewsnet.atlassian.net/browse/DATA-784Fix price charts located under the “Old Charts” tab - https://fewsnet.atlassian.net/browse/DATA-560

• Investigate potential templates (auto-complete field) loading issue with the new Autocomplete functionality - https://fewsnet.atlassian.net/browse/DATA-957

• Create_sample_data doesn’t work on production containers MarketProduct search doesn’t recognize partial Price Types - https://fewsnet.atlassian.net/browse/DATA-992961

• Error extracting Haiti and Madagascar availability data - https://fewsnet.atlassian.net/browse/DATA-1000 MarketProduct search doesn’t recognize partial

• Price Types Collection date field is not automatically populated on the data collection page - https://fewsnet.atlassian.net/browse/DATA-961Zimbabwe common currency prices not being calculated in data extracts 1002

Features

Investigate dissolve performance in /api/ipcpasemap - https://fewsnet.atlassian.net/browse/DATA-

...

Investigate potential templates (auto-complete field) loading issue with the new Autocomplete functionality - https://fewsnet.atlassian.net/browse/DATA-957

...

621

• This feature improved the performance of GeoJSON downloads for map visualizations by dissolving internal boundaries prior to extraction so that the download doesn't need to contain any of the internal boundaries under each phase; for instance:

  • 1) https://fdw.fews.net/api/ipcphasemap/?format=geojson&country_code=MZ&collection_date=2020-10-01&scenario=ML1&_refresh=true

  • 2) https://fdw.fews.net/api/ipcphasemap/?format=geojson&country_code=NG&collection_date=2020-10-01&scenario=ML1&_refresh=true

  • 3) https://fdw.fews.net/api/ipcphasemap/?format=geojson&country_code=MG&collection_date=2020-10-01&scenario=ML1&_refresh=true.

Replace Staple Food Price Data Set on Data center with direct download from FDW - https://fewsnet.atlassian.net/browse/DATA-

...

906

• This implementation allows the direct download of the price data file from the data center on the FEWS NET website (https://fewsnetfdw.atlassianfews.net/api/browse/DATA-992

...

• marketprice.xlsx/?dataset=FEWS_NET_Price_Data)

  and the downloaded file contains the selected fields fit for the website extracts.

Review the pre-

...

processing and matching of DataSourceDocuments in uploads - https://fewsnet.atlassian.net/browse/DATA-935

• This feature is about removing trailing spaces in DataSourceDocment namesupdating the data_source_document field with a trailing space to be recognized in an uploading process.

Add collection_start_date and collection_end_date flters to DataPointFilter - https://fewsnet.atlassian.net/browse/DATA-968

• This feature enhancement is about incorporating the collection_start_date and collection_end_date filters to the DataPointFilter , DataCollectionPeriodFilter and DataCollectionFilter which enables users to easily update local data visualizations each month via the API to update visualizations without additional effort by using the ‘collection_start_date’ and ‘collection_end_date’ filters in the Data Point Filter.

...

• , as seen:

  • https://fdw.fews.net/api/ipcphase/?format=geojson&collection_date_start=2021-04-01&country_code=KE , https://fdw.fews.net/api/datacollectionperiod/?format=json&collection_start_date=2021-04-01&data_domain=IPCCLASSIFICATION

  • https://fdw.fews.net/api/datacollection/?format=json&collection_start_date=2021-04-01&data_domain=IPCCLASSIFICATION ).

Support fields parameter for geojson extracts of DataPointViewSet - https://fewsnet.atlassian.net/browse/DATA-

...

969

• This enhancement is about incorporating the fields parameter for GeoJSON extracts of the DataPointViewSet API endpoint, for example: https://fdw.fews.net/api/ipcphase/?format=geojson&collection_date=2020-10-01&country_code=KE&fields=id,scenario,start_date,end_date,collection_date,value.

Support As-Is imports on the three usual import screens https://fewsnet.atlassian.net/browse/DATA-

...

971

• This feature improved the performance of GeoJSON downloads for map visualizations by dissolving internal boundaries prior to extraction.

...

• is about implementing the As-Is imports in the three data import screens of the FDW. Users can now upload as-is data (i.e. data that has not been cleaned to match the metadata standards) into the FDW in the same way that data is currently uploaded, for example:

1. Offline Data Uploads screen (https://fdw.fews.net/en/admin/warehouse/dataupload/add/)

...

2. Add Data Collection screen  (https://fdw.fews.net/en/admin/warehouse/datacollection/add/)

...

3. Semi-Structured Data Point Admin Screen (https://fdw.fews.net/en/admin/semistructured/asisdatapoint/import/)

...

As a Food Security Analyst, I want to be able to view the admin 1 & 2 names when I extract market price data https://fewsnet.atlassian.net/browse/DATA-

...

974

• This feature developed standard price charts in superset and applied customization where possible.

...

• is about allowing the inclusions of Admin1 and Admin2 names in the market price data extracts.

As a Food Security Analyst, I want to be able to view the CPI subnational data associated with an admin 1 in my extract https://fewsnet.atlassian.net/browse/DATA-

...

975

• This feature enables the users to download the subset of fields for IPC data required for analysis

...

• enhancement is about showing the CPI subnational data which are linked with Admin1 units, (for instance, https://fdw.fews.net/api/datapointfacts/?format=html&dataseries=26133&fields=simple).

Tasks

Trade Flow resource generates lots of duplicate queries https://fewsnet.atlassian.net/browse/DATA-

...

• This feature creates component tests for react components in FDE2

Tasks

950

• Since the trade flow resource generates multiple queries which will make the data upload process slow, this task was about generating fewer queries which makes the data upload process faster .

Enable extraction of reusable visualizations in JSON format - https://fewsnet.atlassian.net/browse/DATA-993

• This task enables was about allowing the successful extraction of a reusable visualizations as visualization in a JSON object

...

• format, for instance:

  • https://

...

• • fdw.

...

• • fews.net/

...

• This task upgrades Django from version 3.1 to 3.2 which includes features such as:

  • Customizing type of auto-created primary keys

  • Functional indexes

  • New decorators for the admin site

  • Admin theming support

  • Creating reusable aliases for expressions that do not need to be selected but are used for filtering, ordering, or as part of complex expressions

Confirm IPC Node checking notification works in PROD - https://fewsnet.atlassian.net/browse/DATA-988

• This task confirms that the IPC Node Checking notifications are working correctly.

Support to Sudan Farmers API Testing - https://fewsnet.atlassian.net/browse/DATA-897

• This task supports testing of the Sudan farmers API and applies relevant modifications to the FAO pipelines.

...

• • viz/viz_visualizations_price_augmentedpricechart.json?dataseries=314489

  • https://fdw.fews.net/viz/viz_visualizations_price_currentprevioushistoricpricechart.json?dataseries__in=314459&historic_range=2y

  • https://fdw.fews.net/viz/viz_visualizations_price_averagepricechangemap.json?dataseries__marketproduct__product=R01112AA&historic_range=1m&dataseries__geographic_unit__country=SD&period_date=2021-02-28

Bugs

Zimbabwe common currency prices not being calculated in data extracts - https://fewsnet.atlassian.net/browse/DATA-

...

• This task reviews the differences between the base library for Mapbox and the mapping features of Plotly for the FDE2 map component.

Bugs

...

533

• The Zimbabwe common currency prices were not being calculated in data extracts and this was resolved.

Fix price charts located under the “Old Charts” tab - https://fewsnet.atlassian.net/browse/DATA-

...

560

• There was an issue in the augmented price chart where the x-axis label and tab names were incorrect, and this has been updated and resolved.

...

Error uploading ET 2018

...

and 2009 LHZ shapefiles - https://fewsnet.atlassian.net/browse/DATA-784

• Users received an internal server error when attempting to upload ET 2018 and 2009 LHZ shapefiles. This issue was resolved.

Fix price charts located under the “Old Charts” tab - https://fewsnet.atlassian.net/browse/DATA-560

• X-Axis labels, chart styles and tab names required updates on the Old Charts tab. This issue was resolved.

Investigate potential templates (auto-complete field) loading issue with the new Autocomplete functionality - https://fewsnet.atlassian.net/browse/DATA-957

• Users experienced loading issues and were unable to select the data source organization while in the data source document add form. This issue was resolved. https://fdwdevThere was a problem in selecting data values for some pages of the FDW, for instance:

  • https://fdw.fews.net/en/admin/warehouse/datacollection/add/

  • https://fdw.fews.net/en/admin/price/marketproduct/add/

  • https://fdw.fews.net/en/admin/

  warehouse

  • price/

  datasourcedocument

  • exchangerate/add/

...

• • https://

...

• • fdw.

...

• • fews.net

...

• • /en/admin/price/priceindex/add/

  • https://

...

• • fdw.

...

• • fews.net/

...

• Users received an error message when attempting to extract availability data for Haiti and Madagascar in FDW. This issue was resolved

  • en/admin/warehouse/datasourcedocument/add/
    This was due to the implementation of the new autocomplete functionality and this was resolved and all the pages are working as expected.

MarketProduct search doesn’t recognize partial Price Types - https://fewsnet.atlassian.net/browse/DATA-961

• Abbreviated price types were not recognized in Market Product searches. This issue was resolved.

...

• There was a problem in the /api/marketproduct/ end point in making search with a partial terms of the price_type , for instance with whol or ret . This was resolved and can be searched using these partial terms, such as https://fdw.fews.net/api/marketproduct/?format=json&search=eth mai ret or
  https://fdw.fews.net/api/marketproduct/?format=json&search=nair mai whol.

Error extracting Haiti and Madagascar availability data - https://fewsnet.atlassian.net/browse/DATA-

...

1000

• There was a problem in extracting the availability data of the countries Haiti and Madagascar and this was resolved.

Collection date field is not automatically populated on the data collection page - https://fewsnet.atlassian.net/browse/DATA-1002

• In FDW, the Collection Date The collection_date field was not automatically populated on the Add Data Collection page. This issue was resolved. populated automatically, which it should have been, while uploading data on the data collection entry page (https://fdw.fews.net/en/admin/warehouse/datacollection/add/

  Image Removed

  ). This was resolved and the functionality is now working as expected.