TouchTerrain standalone in a jupyter notebook

Chris Harding, Aug. 27, 2019

  • this jupyter notebook runs a standalone version of TouchTerrain, similar to TouchTerrain_standalone.py
  • this notebook needs to be run in Python 3.x (I'm using 3.7)
  • I assume you've installed anaconda or miniconda and have the standard 3.party packages already install (e.g. numpy, pillow (aka PIL), etc.)
  • the following additional packages are required, either use (ana)conda or pip to install:
    • gdal (On windows, if conda gives you trouble, get the whl file from here and install via pip install <whl file>
    • vectors: install via pip install vectors
    • earthengine-api (ONLY if you want to use Google Earth Engine's (GEE) online DEM data, NOT needed for converting local DEM raster files (geotiffs)
  • if you're never going to use Google Earth Engine's (GEE) online DEM data skip the next section and go to Running TouchTerrain

Using Google Earth Engine online DEM rasters

  • You don't need this if you only want to import terrain from locally stored raster files (geotiffs)!
  • TouchTerrain can use DEM data from Google Earth Engine (given the corners of the area), but you need to first request a developer account and set up an authentication file
  • (This dev account is different from your standard Google (Gmail, etc.) account!)
  • Getting the account is free and entitles you to a modest number of requests (4 per seconds), which are also free. To request got to https://signup.earthengine.google.com/, you'll get and email with a file.
  • refer to the part Setting Up Authentication Credentials https://developers.google.com/earth-engine/python_install_manual (ignore the stuff above it, as you should already have installed all the needed packages when you installed the earthengine-api package ...)
In [1]:
# comment out (remove the #) the next line and run this cell to see if you could use ee (Earth Engine)
#import ee; ee.Initialize()

if you get:

Please authorize access to your Earth Engine account by running earthengine authenticate in your command line, and then retry.

comment out and run the cell below, the ! means it will run earthengine authenticate inside a OS shell (commandline).

  • This will asked to to sign in to your Google Account and give you a very long authentication token.
  • Paste that token in, to your commandline
  • if correct it will create a folder .config in your home folder and create a authentication file (in a earthengine folder)
  • with this file in place go back to the cell above (import ee;ee.Initialize()) and run it
  • this time, it should work and you now have access to the Google Earth engine API and it's terrain data (DEM sources)
In [2]:
#!earthengine authenticate

Running TouchTerrain

Click on the cell below and hit Shift-Enter to run it (won't show anything, just does some path setup stuff)

In [4]:
# RUN THIS CELL TO START

# just some setup stuff - nothing to see here
import os, sys
from os.path import abspath
from pprint import pprint
#from glob import glob

# need to add parent folder to sys.path, so we can later import common, which is a sibling to standalne
this_folder = abspath(os.getcwd())
parentfolder = abspath(os.getcwd() + os.sep + "..")
sys.path.append(parentfolder)
#print(sys.path)
print('setup done')
setup done

Put your settings into the dictionary below and hit Shift-Enter

  • for more info on the settings, look at the ReadMe on https://github.com/ChHarding/TouchTerrain_for_CAGEO
  • note, however, that the settings given below are in Python syntax, whereas the ReadMe describes the JSON syntax used in the config file
  • both are very similar except for None and True/False
  • Python vs JSON:
    • None null
    • True true
    • False false
In [5]:
args = {
    # DEM/Area to print
    
    # A: use local DEM raster (geotiff)
    #"importedDEM": "pyramid.tif",  # put file in same folder as this notebook file!
    
    # B: use area and a DEM online source via EarthEngine
    "importedDEM": None,
    "DEM_name": "USGS/NED",   # DEM source
    "bllat": 44.50185267072875,   # bottom left corner lat
    "bllon": -108.25427910156247, # bottom left corner long
    "trlat": 44.69741706507476,   # top right corner lat
    "trlon": -107.97962089843747, # top right corner long
    
    # 3D print parameters
    "tilewidth": 80,  # width of each tile in mm, (tile height will be auto calculated)
    "printres": 0.4,  # resolution (horizontal) of 3D printer (= size of one pixel) in mm, 
                      # should be your nozzle size or just a bit less! 
                      # Using something like 0.01 will NOT print out a super detailed version 
                      # as you slicer will remove such fine details anyway! Instead, you'll
                      # just wait a long time and get a super large STL file!
                      # If you want the original resolution of the DEM, use -1
    
    "ntilesx": 1, # number of tiles in x  
    "ntilesy": 1, # number of tiles in y    

    "basethick": 0.5,   # thickness (in mm) of printed base
    "zscale": 3,      # elevation (vertical) scaling
    "fileformat": "STLb",  # format of 3D model files: "obj" wavefront obj (ascii),
                           #   "STLa" ascii STL or "STLb" binary STL.
                           #   To export just the (untiled) raster (no mesh), use "GeoTiff" 
    "zip_file_name": "myterrain",   # base name of zipfile, .zip will be added

    
    # Expert settings
    "tile_centered": False, # True-> all tiles are centered around 0/0, False, all tiles "fit together"
    "CPU_cores_to_use" : 0, # 0: use all available cores, None: don't use multiprocessing (single core only)
                            # multi-core will be much faster for more than 1 tile 
    "max_cells_for_memory_only" : 5000^2, # if number of raster cells is bigger than this, use temp_files instead of memory.
                            # set this very high to force use of memory and lower it if you run out of memory
    "no_bottom": False,   # omit bottom triangles? Most slicers still work and it makes smaller files
    "no_normal": True,    # Don't calculate normals for triangles. This is significantly faster but some viewer may need them.
    "bottom_image": None, # 1 band greyscale image used for bottom relief
    "ignore_leq": None,   # set all values <= this to NaN so they don't print
    "unprojected": False, # don't project to UTM (for EE rasters only)
    "projection": None,   # None means use the closest UTM zone. Can be a EPSG number (int!) instead but not all work. 
    "only" : None,        # if not None: list with x and y tile index (1 based) of the only tile to process
                          #   e.g. [1,1] will only process the tile in upper left corner, [2,1] the tile right to it, etc.
}

########################################################

# if we want to work on a local raster, get the full pathname to it
if args["importedDEM"] != None: 
    args["importedDEM"]= abspath(args["importedDEM"]) 
#pprint(args)
print("settings stored, ready to process")
settings stored, ready to process
In [10]:
from common import TouchTerrainEarthEngine as TouchTerrain
# Process the data

# This may take some time! You'll see In[*] and some log messages (will also be in the logfile inside the zip)
# You may see some red stuff with 10%, etc. - don't worry, that's normal
totalsize, full_zip_file_name = TouchTerrain.get_zipped_tiles(**args) # all args are in a dict
print("\nDONE!\n\nCreated zip file", full_zip_file_name,  "%.2f" % totalsize, "Mb")

# your zip file will be inside the tmp folder which is inside the same folder your notebook file is in
Log for creating 3D model tile(s) for  NED_-108.12_44.60 
 
DEM_name = USGS/NED 
trlat = 44.69741706507476 
trlon = -107.97962089843747 
bllat = 44.50185267072875 
bllon = -108.25427910156247 
printres = 0.4 
ntilesx = 1 
ntilesy = 1 
tilewidth = 80 
basethick = 0 
zscale = 3 
fileformat = STLb 
no_bottom = False 
unprojected = False 
no_normals = True 

process started: 12:54:00.921694 

Region (lat/lon):
   44.69741706507476 -107.97962089843747 (top right)
   44.50185267072875 -108.25427910156247 (bottom left) 
center at [-108.11694999999997, 44.599634867901756]  UTM 13 N ,  EPSG:32613 
lon/lat size in degrees: [0.274658203125, 0.19556439434600748] 
requesting 109.03102605841676 m resolution from EarthEngine
Earth Engine raster: USGS/NED 
 USGS National Elevation Dataset 1/3 arc-second 
 geotiff size: 0.16556549072265625 Mb 
 cell size 109.03102605841676 m, upper left corner (x/y):  241285.66066727627 4954587.886146574 
full (untiled) raster (height,width):  (208, 208) float32 
cell size: 109.03102605841676 m  
adjusted print res from the requested 0.4 mm to 0.38461538461538464 mm to ensure correct model dimensions 
total model size in mm: 80 x 80.0 
map scale is 1 : 283480.66775188356 
elev min/max : 1125.44 to 1578.55
Cells per tile (x/y) 208 x 208 
using single-core only 
processing tile: 1 1
top min/max: 0.5 14.885345
creating internal triangle data structure for <_MainProcess(MainProcess, started)>
10 % <_MainProcess(MainProcess, started)>
20 % <_MainProcess(MainProcess, started)>
30 % <_MainProcess(MainProcess, started)>
40 % <_MainProcess(MainProcess, started)>
50 % <_MainProcess(MainProcess, started)>
60 % <_MainProcess(MainProcess, started)>
70 % <_MainProcess(MainProcess, started)>
80 % <_MainProcess(MainProcess, started)>
90 % <_MainProcess(MainProcess, started)>
100% <_MainProcess(MainProcess, started)> 

Writing tile into temp. file C:\Users\charding\Box\TouchTerrain\TouchTerrain_chris_dev3.1\standalone\tmp\myterrain11.tmp
assembling binary stl from 174720 triangles
10 %, 20 %, 30 %, 40 %, 50 %, 60 %, 70 %, 80 %, 90 %, 100 %, 

tile 1 1 STLb 8.331378936767578 Mb 
1 x 1 tiles, tile size 80.00 x 80.00 mm
 
tile 1 1 : height:  0.5 - 14.885345 mm , file size: 8 Mb 

total size for all tiles: 8 Mb 
zip finished: 12:54:06.806752
added full geotiff as NED_-108.12_44.60.tif 

processing finished: 12:54:06.818758 

DONE!

Created zip file tmp\myterrain.zip 8.50 Mb
In [11]:
# If you want to unzip the zip file, run this cell
# (You will need to do this before using k3d for visualization)

import os.path
from glob import glob
folder, file = os.path.splitext(full_zip_file_name) # get folder of zip file

# unzip the zipfile into the folder it's already in
import zipfile
zip_ref = zipfile.ZipFile(full_zip_file_name, 'r')
zip_ref.extractall(folder)
zip_ref.close()
print ("unzipped files from", full_zip_file_name, "into the folder", folder)
print (folder, "contains these files:")
for f in glob(folder + os.sep + "*.*"): print(" ", f)
unzipped files from tmp\myterrain.zip into the folder tmp\myterrain
tmp\myterrain contains these files:
  tmp\myterrain\logfile.txt
  tmp\myterrain\NED_-108.12_44.60.tif
  tmp\myterrain\NED_-108.12_44.60_tile_1_1.STL

Visualize the STL file(s)

  • If you want to visualize your model(s), install k3d (pip install k3d) and run the cell below
  • this won't work with OBJ files, as k3d can't read them in
In [12]:
import k3d

# get all stl files in that folder
mesh_files = glob(folder + os.sep + "*.stl")
#print "in folder", folder, "using", mesh_files

plot = k3d.plot()

from random import randint
for m in mesh_files:
    col = (randint(0,255) << 16) + (randint(0,255) << 8) + randint(0,255) # random rgb color as hex
    print("adding to viewer:", m, hex(col))
    buf = open(m, 'rb').read()
    plot += k3d.stl(buf, color=col)
plot.display()
adding to viewer: tmp\myterrain\NED_-108.12_44.60_tile_1_1.STL 0x19338f
In [ ]: