Difference between revisions of "How to convert data for Geogrid"

From openwfm
Jump to navigation Jump to search
 
(32 intermediate revisions by 4 users not shown)
Line 1: Line 1:
−
Running WRF-Fire using real data requires additional datasets not included in the standard
+
{{users guide}}
−
WPS input data [http://www.mmm.ucar.edu/wrf/src/wps_files/geog_v3.1.tar.gz tarball].  For these,
 
−
it is necessary to convert your source data into the
 
−
[http://www.mmm.ucar.edu/wrf/users/docs/user_guide_V3.1/users_guide_chap3.htm#_Writing_Static_Data geogrid data format].
 
−
The geogrid data format consists of a directory of tiled binary files with names indicating the index range contained in each tile.
 
−
For instance, a file name of <tt>00101-00200.00051-00100</tt> would consist of columns 101 through 200 and rows 51 through 100.
 
−
The files themselves contain only an array integers, with no associated metadata.
 
−
The metadata for the dataset is contained in a file called <tt>index</tt>, which contains a series ''keyword=value'' statements
 
−
telling geogrid the details of the data tiles, such as width, height, the number of bytes per integer, etc.  The index file also contains metadata
 
−
for the data set itself, such as the projection information and physical units.
 
  
−
We have made available a series of scripts and utilities ([http://github.com/jbeezley/autoWPS autoWPS]) intended to automate many of the steps
+
This wiki page explains how to transform geotiff files to geogrid using convert_geotiff script in [https://github.com/openwfm/wrfxpy.git wrfxpy].
−
required to create a dataset for a WRF real run. A utility program included in this repository, WPSGeoTiff, is a simple command line utility that
 
−
takes a GeoTiff file and writes converts the data into geogrid binary format.  The GeoTiff format was chosen because it is a highly ubiquitous format
 
−
for geotagged data and it has a cross-platform and open source c library called [http://trac.osgeo.org/geotiff/ libGeoTIFF].  Most, if not all, GIS application
 
−
software is capable of converting or exporting a dataset as a GeoTiff file.
 
  
−
=Prerequisites=
+
== Get Anaconda3 distribution ==
  
−
The WPSGeoTiff utility requires two external libraries in order to compile.  You should pick a location to install the libraries, for instance <tt>PREFIX=${HOME}/opt</tt>.
+
Download and install the Python 3 [https://www.anaconda.com/products/individual Anaconda Python] distribution for your platform. We recommend an installation into the user's home directory.
−
* [http://www.libtiff.org/ libTIFF] The following commands are sufficient to install libTIFF for most systems.
 
−
<code>./configure --prefix=$PREFIX
 
−
make
 
−
make install</code>
 
−
* [http://trac.osgeo.org/geotiff/ libGeoTIFF] A similar set of commands install libGeoTIFF, but we must tell the configure script were we put libTIFF.
 
−
<code>./configure --prefix=$PREFIX --with-libtiff=$PREFIX
 
−
make
 
−
make install</code>
 
  
−
=Obtaining the WPSGeoTiff source=
+
== Install environment ==
  
−
The autoWPS package is available as a git repository hosted at [http://github.com/jbeezley/autoWPS github].  The repository can be cloned with the
+
Create the python environment:
−
[http://git-scm.com/ git] command:
 
−
<code>git clone git://github.com/jbeezley/autoWPS.git</code>
 
−
A tarball of the latest release can also be obtained
 
−
at [http://github.com/jbeezley/autoWPS/tarball/master http://github.com/jbeezley/autoWPS/tarball/master].
 
  
−
=Compiling WPSGeoTiff=
+
<pre>
 +
conda update -n base -c defaults conda
 +
conda create -n wrfx python=3 gdal netcdf4 pyproj paramiko dill h5py psutil proj4 pytz flask pandas
 +
conda activate wrfx
 +
conda install -c conda-forge simplekml pygrib f90nml pyhdf xmltodict basemap rasterio scipy
 +
pip install MesoPy python-cmr
 +
</pre>
  
−
Inside the main directory of the autoWPS release is a subdirectory called WPSGeoTiff.  This utility is written in pure c and creates a standalone binary.
+
Note that <tt>conda</tt> and <tt>pip</tt> are package managers available in the Anaconda Python distribution.
−
In most cases it can be compiled by issuing the following command.
 
−
<code>LIBTIFF=$PREFIX GEOTIFF=$PREFIX make</code>
 
−
The main binary <tt>convert_geotiff.x</tt>, should now reside in the current directory.
 
  
−
=Running WPSGeoTiff=
+
== Get wrfxpy repository ==
  
−
Running <tt>convert_geotiff.x</tt> with no arguments will produce a usage description.
+
Get repository wrfxpy and checkout to convert_geotiff branch. Shortly, it is going to be available in the master branch.
−
<code><pre>Usage: ./convert_geotiff.x [OPTIONS] FileName
+
git clone https://github.com/openwfm/wrfxpy
 +
cd wrfxpy
 +
git checkout convert_geotiff
  
−
Converts geotiff file `FileName' into geogrid binary format
+
== Run convert_geotiff.sh ==
−
into the current directory.
 
  
−
Options:
+
Run the code to generate geogrid from geotiff files. Run the following line for each variable to process.
−
-h        : Show this help message and exit
+
  ./convert_geotiff.sh geotiff_file geo_data_path wrf_variable_name [bbox]
−
-c NUM    : Indicates categorical data (NUM = number of categories)
 
−
-b NUM    : Tile border width (default 3)
 
−
-w [1,2,4] : Word size in output in bytes (default 2)
 
−
-z        : Indicates unsigned data (default FALSE)
 
−
-t NUM    : Output tile size (default 100)
 
−
-s SCALE  : Scale factor in output (default 1.)
 
−
-m MISSING : Missing value in output (default 0., ignored for categorical data)
 
−
-u UNITS  : Units of the data (default "NO UNITS")
 
−
-d DESC    : Description of data set (default "NO DESCRIPTION")
 
−
</pre></code>
 
−
All of the files will be created in the current directory, so it is best to run the program from an empty directory. A more detailed description of the  
 
−
arguments to this program follows.
 
−
*; <tt>-b</tt>
 
−
:<p>The data tiles in the geogrid binary format are allowed to overlap by a fixed number of grid points.  The extra border around the tile is called the halo, and this argument sets the width of the halo.  For instance with a halo of size three, the file named <tt>00101-00200.00051-00100</tt> would actually contain columns 98-203 and rows 48-103 of the full dataset.  This halo is necessary for the interpolation scheme inside of WPS.  The default should be acceptable for most situations.</p>
 
−
*; <tt>-w</tt>
 
−
:<p>The number of bytes to represent each data point as an integer.  These integers are scaled by the scaling parameter before being truncated to an integer. scaledA lower value will make the output data smaller, at the cost of accuracy or the dynamic range of the input.</p>
 
−
*; <tt> -m</tt>
 
−
:<p>Any grid point that is missing data, such as the outer border of the edge tiles, or grid points that the GeoTIFF file indicates as missing will be set to this value. This argument is currently ignored when the categorical flag is set, instead missing data will be set to the maximum category + 1.</p>
 
−
*;<tt>-s</tt>
 
−
:<p>Because the data is always stored as an integer, a scaling parameter is needed to represent fractional numbers or large values. The data set will be divided by this number prior to being truncated to an integer.  If the data set has an accuracy of 2 decimal places, a reasonable scale to use would be 0.01.</p>
 
−
*;<tt>-u, -d</tt>
 
−
:<p>The units and a small description of the data set should be included as arguments.  Multi-word arguments should be quoted as follows. <code><pre>-u meters -d "elevation above sea level"</pre></code></p>
 
−
*; <tt>FileName</tt>
 
−
:<p>The final argument must always be present.  This is the (absolute or relative) path to the GeoTIFF file to be converted.</p>
 
  
−
=Limitations=
+
where:
  
−
The current code has several limitations which are listed here.
+
* <tt>geotiff_file</tt> - path to a geotiff file to be converted to geogrid
−
* Datasets must not contain more than 99,999 grid points in each axis. This is a limitation of the geogrid format itself, due to the naming convention of the tiles.  However, it is possible (but inefficient) to split a single dataset into multiple directories for this purpose. A better solution would be to resample the data to a lower spatial resolution prior to converting.
+
* <tt>geo_data_path</tt> - any path where to save all the variables
−
* This program cannot convert between geographic projections, so the input data must be in a projection supported by WPS. All of the projections [http://www.mmm.ucar.edu/wrf/users/docs/user_guide_V3.1/users_guide_chap3.htm#_Description_of_index supported by WPS] should work for this conversion program; however, only UTM, Albers equal area, and lat-lon have been tested.  In addition, data sources may not conform to [http://www.spatialreference.org/ EPSG standards] in their projection tags; the output should always be checked before use.
+
* <tt>wrf_variable</tt> - WRF variable created on src/geo/var_wisdom.py
 +
* <tt>bbox</tt> - optional, bounding box to sample the original geotiff file from, using WGS84 coordinates. Format: min_lon,max_lon,min_lat,max_lat
 +
 
 +
If you want to know more information and the existing options for wrf_variable, run ./convert_geotiff.sh without inputs
 +
./convert_geotiff.sh
 +
 
 +
Some examples:
 +
./convert_geotiff.sh elevation.tif geo_data ZSF
 +
./convert_geotiff.sh fuel.tif geo_data NFUEL_CAT
 +
  ./convert_geotiff.sh fuel.tif geo_data NFUEL_CAT -112.8115,-112.1661,39.4820,39.9750
 +
 
 +
== Results from convert_geotiff.sh ==
 +
 
 +
The convert_geotiff.sh script will generate a folder specified by the user by <tt>geo_data_path</tt> with
 +
 
 +
* '''Variable folders:''' A folder for each variable previously run (ZSF and NFUEL_CAT folders) containing:
 +
** a geogrid file of the variable.
 +
** an index file of the variable.
 +
* '''geogrid_tbl.json:''' A JSON file with all the geogrid information for each variable.
 +
* '''GEOGRID.TBL:''' A text file with the information to add to GEOGRID.TBL (depends on src/geo/var_wisdom.py, so only copy what you need).
 +
* '''index.json:''' A JSON file with index information for each variable.
 +
 
 +
[[Category:WRF-Fire]]
 +
[[Category:Howtos|Convert data for Geogrid]]
 +
[[Category:Data]]

Latest revision as of 01:59, 17 February 2021

Back to the WRF-SFIRE user guide.

This wiki page explains how to transform geotiff files to geogrid using convert_geotiff script in wrfxpy.

Get Anaconda3 distribution

Download and install the Python 3 Anaconda Python distribution for your platform. We recommend an installation into the user's home directory.

Install environment

Create the python environment: 

 conda update -n base -c defaults conda
 conda create -n wrfx python=3 gdal netcdf4 pyproj paramiko dill h5py psutil proj4 pytz flask pandas
 conda activate wrfx
 conda install -c conda-forge simplekml pygrib f90nml pyhdf xmltodict basemap rasterio scipy
 pip install MesoPy python-cmr

Note that conda and pip are package managers available in the Anaconda Python distribution.

Get wrfxpy repository

Get repository wrfxpy and checkout to convert_geotiff branch. Shortly, it is going to be available in the master branch. 

git clone https://github.com/openwfm/wrfxpy
cd wrfxpy
git checkout convert_geotiff

Run convert_geotiff.sh

Run the code to generate geogrid from geotiff files. Run the following line for each variable to process. 

./convert_geotiff.sh geotiff_file geo_data_path wrf_variable_name [bbox]

where:

  • geotiff_file - path to a geotiff file to be converted to geogrid
  • geo_data_path - any path where to save all the variables
  • wrf_variable - WRF variable created on src/geo/var_wisdom.py
  • bbox - optional, bounding box to sample the original geotiff file from, using WGS84 coordinates. Format: min_lon,max_lon,min_lat,max_lat

If you want to know more information and the existing options for wrf_variable, run ./convert_geotiff.sh without inputs 

./convert_geotiff.sh

Some examples: 

./convert_geotiff.sh elevation.tif geo_data ZSF
./convert_geotiff.sh fuel.tif geo_data NFUEL_CAT
./convert_geotiff.sh fuel.tif geo_data NFUEL_CAT -112.8115,-112.1661,39.4820,39.9750

Results from convert_geotiff.sh

The convert_geotiff.sh script will generate a folder specified by the user by geo_data_path with

  • Variable folders: A folder for each variable previously run (ZSF and NFUEL_CAT folders) containing:
    • a geogrid file of the variable.
    • an index file of the variable.
  • geogrid_tbl.json: A JSON file with all the geogrid information for each variable.
  • GEOGRID.TBL: A text file with the information to add to GEOGRID.TBL (depends on src/geo/var_wisdom.py, so only copy what you need).
  • index.json: A JSON file with index information for each variable.
Retrieved from "http://wiki.openwfm.org/mediawiki/index.php?title=How_to_convert_data_for_Geogrid&oldid=4333"