Difference between revisions of "How to convert data for Geogrid"

From openwfm
Jump to navigation Jump to search
 
(36 intermediate revisions by 4 users not shown)
Line 1: Line 1:
−
Running WRF-Fire using real data requires additional datasets not included in the standard
+
{{users guide}}
−
WPS input data [http://www.mmm.ucar.edu/wrf/src/wps_files/geog_v3.1.tar.gz tarball].  For these,
 
−
it is necessary to convert your source data into the
 
−
[http://www.mmm.ucar.edu/wrf/users/docs/user_guide_V3.1/users_guide_chap3.htm#_Writing_Static_Data geogrid data format].
 
−
The geogrid data format consists of a directory of tiled binary files with names indicating the index range contained in each tile.
 
−
For instance, a file name of <tt>00101-00200.00051-00100</tt> would consist of columns 101 through 200 and rows 51 through 100.
 
−
The files themselves contain only an array integers, with no associated metadata.
 
−
The metadata for the dataset is contained in a file called <tt>index</tt>, which contains a series ''keyword=value'' statements
 
−
telling geogrid the details of the data tiles, such as width, height, the number of bytes per integer, etc.  The index file also contains metadata
 
−
for the data set itself, such as the projection information and physical units.
 
  
−
We have made available a series of scripts and utilities ([http://github.com/jbeezley/autoWPS autoWPS]) intended to automate many of the steps
+
This wiki page explains how to transform geotiff files to geogrid using convert_geotiff script in [https://github.com/openwfm/wrfxpy.git wrfxpy].
−
required to create a dataset for a WRF real run. A utility program included in this repository, WPSGeoTiff, is a simple command line utility that
 
−
takes a GeoTiff file and writes converts the data into geogrid binary format.  The GeoTiff format was chosen because it is a highly ubiquitous format
 
−
for geotagged data and it has a cross-platform and open source c library called [http://trac.osgeo.org/geotiff/ libGeoTIFF].  Most, if not all, GIS application
 
−
software is capable of converting or exporting a dataset as a GeoTiff file.
 
  
−
=Prerequisites=
+
== Get Anaconda3 distribution ==
  
−
The WPSGeoTiff utility requires two external libraries in order to compile.  You should pick a location to install the libraries, for instance <tt>PREFIX=${HOME}/opt</tt>.
+
Download and install the Python 3 [https://www.anaconda.com/products/individual Anaconda Python] distribution for your platform. We recommend an installation into the user's home directory.
−
* [http://www.libtiff.org/ libTIFF] The following commands are sufficient to install libTIFF for most systems.
 
−
<code>./configure --prefix=$PREFIX
 
−
make
 
−
make install</code>
 
−
* [http://trac.osgeo.org/geotiff/ libGeoTIFF] A similar set of commands install libGeoTIFF, but we must tell the configure script were we put libTIFF.
 
−
<code>./configure --prefix=$PREFIX --with-libtiff=$PREFIX
 
−
make
 
−
make install</code>
 
  
−
=Obtaining the WPSGeoTiff source=
+
== Install environment ==
  
−
The autoWPS package is available as a git repository hosted at [http://github.com/jbeezley/autoWPS github].  The repository can be cloned with the
+
Create the python environment:
−
[http://git-scm.com/ git] command:
 
−
<code>git clone git://github.com/jbeezley/autoWPS.git</code>
 
−
A tarball of the latest release can also be obtained
 
−
at [http://github.com/jbeezley/autoWPS/tarball/master http://github.com/jbeezley/autoWPS/tarball/master].
 
  
−
=Compiling WPSGeoTiff=
+
<pre>
 +
conda update -n base -c defaults conda
 +
conda create -n wrfx python=3 gdal netcdf4 pyproj paramiko dill h5py psutil proj4 pytz flask pandas
 +
conda activate wrfx
 +
conda install -c conda-forge simplekml pygrib f90nml pyhdf xmltodict basemap rasterio scipy
 +
pip install MesoPy python-cmr
 +
</pre>
  
−
Inside the main directory of the autoWPS release is a subdirectory called WPSGeoTiff.  This utility is written in pure c and creates a standalone binary.
+
Note that <tt>conda</tt> and <tt>pip</tt> are package managers available in the Anaconda Python distribution.
−
In most cases it can be compiled by issuing the following command.
 
−
<code>LIBTIFF=$PREFIX GEOTIFF=$PREFIX make</code>
 
−
The main binary <tt>convert_geotiff.x</tt>, should now reside in the current directory.
 
  
−
=Running WPSGeoTiff=
+
== Get wrfxpy repository ==
  
−
Running <tt>convert_geotiff.x</tt> with no arguments will produce a usage description.
+
Get repository wrfxpy and checkout to convert_geotiff branch. Shortly, it is going to be available in the master branch.
−
<code><pre>Usage: ./convert_geotiff.x [OPTIONS] FileName
+
git clone https://github.com/openwfm/wrfxpy
 +
cd wrfxpy
 +
git checkout convert_geotiff
  
−
Converts geotiff file `FileName' into geogrid binary format
+
== Run convert_geotiff.sh ==
−
into the current directory.
 
  
−
Options:
+
Run the code to generate geogrid from geotiff files. Run the following line for each variable to process.
−
-h        : Show this help message and exit
+
./convert_geotiff.sh geotiff_file geo_data_path wrf_variable_name [bbox]
−
-c NUM    : Indicates categorical data (NUM = number of categories)
+
 
−
-b NUM    : Tile border width (default 3)
+
where:
−
-w [1,2,4] : Word size in output in bytes (default 2)
+
 
−
-z        : Indicates unsigned data (default FALSE)
+
* <tt>geotiff_file</tt> - path to a geotiff file to be converted to geogrid
−
-t NUM    : Output tile size (default 100)
+
* <tt>geo_data_path</tt> - any path where to save all the variables
−
-s SCALE  : Scale factor in output (default 1.)
+
* <tt>wrf_variable</tt> - WRF variable created on src/geo/var_wisdom.py
−
-m MISSING : Missing value in output (default 0., ignored for categorical data)
+
* <tt>bbox</tt> - optional, bounding box to sample the original geotiff file from, using WGS84 coordinates. Format: min_lon,max_lon,min_lat,max_lat
−
-u UNITS  : Units of the data (default "NO UNITS")
+
 
−
-d DESC    : Description of data set (default "NO DESCRIPTION")
+
If you want to know more information and the existing options for wrf_variable, run ./convert_geotiff.sh without inputs
−
</pre></code>
+
./convert_geotiff.sh
−
All of the files will be created in the current directory, so it is best to run the program from an empty directory. A more detailed description of the  
+
 
−
arguments to this program follows.
+
Some examples:
−
*; <tt>-b</tt>
+
./convert_geotiff.sh elevation.tif geo_data ZSF
−
:<p>The data tiles in the geogrid binary format are allowed to overlap by a fixed number of grid points. The extra border around the tile is called the halo, and this argument sets the width of the halo. For instance with a halo of size three, the file named <tt>00101-00200.00051-00100</tt> would actually contain columns 98-203 and rows 48-103 of the full dataset. This halo is necessary for the interpolation scheme inside of WPS. The default should be acceptable for most situations.</p>
+
./convert_geotiff.sh fuel.tif geo_data NFUEL_CAT
 +
./convert_geotiff.sh fuel.tif geo_data NFUEL_CAT -112.8115,-112.1661,39.4820,39.9750
 +
 
 +
== Results from convert_geotiff.sh ==
 +
 
 +
The convert_geotiff.sh script will generate a folder specified by the user by <tt>geo_data_path</tt> with
 +
 
 +
* '''Variable folders:''' A folder for each variable previously run (ZSF and NFUEL_CAT folders) containing:  
 +
** a geogrid file of the variable.
 +
** an index file of the variable.
 +
* '''geogrid_tbl.json:''' A JSON file with all the geogrid information for each variable.
 +
* '''GEOGRID.TBL:''' A text file with the information to add to GEOGRID.TBL (depends on src/geo/var_wisdom.py, so only copy what you need).
 +
* '''index.json:''' A JSON file with index information for each variable.
 +
 
 +
[[Category:WRF-Fire]]
 +
[[Category:Howtos|Convert data for Geogrid]]
 +
[[Category:Data]]

Latest revision as of 01:59, 17 February 2021

Back to the WRF-SFIRE user guide.

This wiki page explains how to transform geotiff files to geogrid using convert_geotiff script in wrfxpy.

Get Anaconda3 distribution

Download and install the Python 3 Anaconda Python distribution for your platform. We recommend an installation into the user's home directory.

Install environment

Create the python environment: 

 conda update -n base -c defaults conda
 conda create -n wrfx python=3 gdal netcdf4 pyproj paramiko dill h5py psutil proj4 pytz flask pandas
 conda activate wrfx
 conda install -c conda-forge simplekml pygrib f90nml pyhdf xmltodict basemap rasterio scipy
 pip install MesoPy python-cmr

Note that conda and pip are package managers available in the Anaconda Python distribution.

Get wrfxpy repository

Get repository wrfxpy and checkout to convert_geotiff branch. Shortly, it is going to be available in the master branch. 

git clone https://github.com/openwfm/wrfxpy
cd wrfxpy
git checkout convert_geotiff

Run convert_geotiff.sh

Run the code to generate geogrid from geotiff files. Run the following line for each variable to process. 

./convert_geotiff.sh geotiff_file geo_data_path wrf_variable_name [bbox]

where:

  • geotiff_file - path to a geotiff file to be converted to geogrid
  • geo_data_path - any path where to save all the variables
  • wrf_variable - WRF variable created on src/geo/var_wisdom.py
  • bbox - optional, bounding box to sample the original geotiff file from, using WGS84 coordinates. Format: min_lon,max_lon,min_lat,max_lat

If you want to know more information and the existing options for wrf_variable, run ./convert_geotiff.sh without inputs 

./convert_geotiff.sh

Some examples: 

./convert_geotiff.sh elevation.tif geo_data ZSF
./convert_geotiff.sh fuel.tif geo_data NFUEL_CAT
./convert_geotiff.sh fuel.tif geo_data NFUEL_CAT -112.8115,-112.1661,39.4820,39.9750

Results from convert_geotiff.sh

The convert_geotiff.sh script will generate a folder specified by the user by geo_data_path with

  • Variable folders: A folder for each variable previously run (ZSF and NFUEL_CAT folders) containing:
    • a geogrid file of the variable.
    • an index file of the variable.
  • geogrid_tbl.json: A JSON file with all the geogrid information for each variable.
  • GEOGRID.TBL: A text file with the information to add to GEOGRID.TBL (depends on src/geo/var_wisdom.py, so only copy what you need).
  • index.json: A JSON file with index information for each variable.
Retrieved from "http://wiki.openwfm.org/mediawiki/index.php?title=How_to_convert_data_for_Geogrid&oldid=4333"