CRX to RNX

CRX is a Hatanaka compression for Rinex file used by many permanent networks. Y. Hatanaka of GSI wrote softwares for compress/uncompress, see https://terras.gsi.go.jp/ja/crx2rnx.html.

To use the software (example of CRX compression to uncompressed RNX format):

  • open a shell (e.g. Powershell)
  • go to the folder containing the executable
  • type “CRX2RNX.exe input.crx”
  • you get as output a RNX file that you can rename to .yyo
  • for help type “CRX2RNX.exe -h”

Use Reach M2 with Phantom 4 Pro

dataset :

photos with timestamps

UBX from M2 and from base station

Processing:

1- Post-Process base station and get corrected coordinates

2- with Emlid Studio (or RTKLib) calculate events (triggers) positions from UBX

as output one gets an …._events.pos file that gives corrected coordinates for each event together with a timestamp

3- extract timestamps from exif of pictures

use exiftool with this command

exiftool -n -csv -filemodifydate -api TimeZone=UTC *.JPG > metadata.csv

the output file “metadata.csv” gives the name of the picture and the timestamp in UTC format

4- Merge …_events.pos and pictures events

For now I’ve got this ugly Python script:

#! /usr/bin/env python

"""
Update Emlid Reach Survey points with PPK position output from RTKLIB
David Shean
dshean@gmail.com
Edited to fix Pandas datetime/Timestamp tz issues, and a few key changes likely based on Emlid updates
"""

import os
import argparse
import numpy as np
import pandas as pd

def getparser():
    parser = argparse.ArgumentParser(description='Update Emlid Reach Survey points with \
            PPK positions from RTKLIB')
    parser.add_argument('survey_pts_csv_fn', type=str, help='Survey point csv filename')
    parser.add_argument('ppk_pos_fn', type=str, help='PPK pos filename')
    return parser


def main():
    parser = getparser()
    args = parser.parse_args()

    survey_pts_csv_fn = args.survey_pts_csv_fn
    ppk_pos_fn = args.ppk_pos_fn

    print('Loading: %s' % survey_pts_csv_fn)
    survey_pts = pd.read_csv(survey_pts_csv_fn, parse_dates=[1], header=0)
    survey_pts['date']=pd.to_datetime(survey_pts['FileModifyDate'],format="%Y:%m:%d %H:%M:%S+00:00")
    survey_pts.sort_values('date', inplace=True)
    survey_pts.index=survey_pts['date']
    print(survey_pts.dtypes)
    print(survey_pts)
    header = 'Date UTC latitude(deg) longitude(deg)  height(m)   Q  ns   sdn(m)   sde(m)   sdu(m)  sdne(m)  sdeu(m)  sdun(m) age(s)  ratio'
    print('Loading: %s' % ppk_pos_fn)
    ppk_pos = pd.read_csv(ppk_pos_fn, comment='%', delim_whitespace=True, names=header.split(), parse_dates=[[0,1]])
    ppk_pos['date']=pd.to_datetime(ppk_pos['Date_UTC'])
    ppk_pos.index=ppk_pos['Date_UTC']
    print(ppk_pos.dtypes)
    print(ppk_pos)

    # Applying merge_asof on data and store it
    # in a variable
    merged_dataframe = pd.merge_asof(ppk_pos, survey_pts, right_index=True,left_index=True,direction='nearest',tolerance=pd.Timedelta("1s"))

    print(merged_dataframe)

    #Write out new file
    out_fn = os.path.splitext(survey_pts_csv_fn)[0]+'_merged.csv'
    print("Writing out: %s" % out_fn)
    merged_dataframe.to_csv(out_fn)    

if __name__ == "__main__":
    main()

5- In Metashape, import the coordinates of the cameras from the new file

Convert raster from ellispoidal height to orthometric altitude

Differences between ellipsoidal height and orthometric altitude (or geoid height) are explained here https://spatialthoughts.com/2019/10/26/convert-between-orthometric-and-ellipsoidal-elevations-using-gdal/

using gdalwarp to convert from UTM33N-WGS84 ellipsoidal height to UTM33N-EGM96 geoid height

gdalwarp -s_srs "+proj=utm +zone=33 +datum=WGS84 +units=m +no_defs" -t_srs "+proj=utm +zone=33 +datum=WGS84 +units=m +no_defs +geoidgrids=egm96_15.gtx" input.tif output.tif

The definition of the projection is the PROJ4 line that can be found on https://spatialreference.org/

Inversely, to convert from altitude to ellipsoidal height use such command (change srs and geoid grid if needed):

gdalwarp -s_srs "+proj=utm +zone=33 +datum=WGS84 +units=m +no_defs +geoidgrids=egm96_15.gtx" -t_srs "+proj=utm +zone=33 +datum=WGS84 +units=m +no_defs" input.tif output.tif

you can choose your input and output spatial reference using the proj4 syntax https://epsg.io/32633 , and note the add of the geoid file with +geoidgrids=egm96_15.gtx in the command.

if you want to use EGM2008 geoid, get the file from https://github.com/OSGeo/proj-datumgrid/blob/master/world/egm08_25.gtx and copy it to the proj directory. In Linux it’s

/usr/share/proj

then you can simply change the geoid file in the command +geoidgrids=egm08_25.gtx

Metashape script

API Python voir https://www.agisoft.com/pdf/metashape_python_api_1_5_0.pdf

FAQ scripting : https://agisoft.freshdesk.com/support/solutions/folders/31000114192

Liste des scripts: https://github.com/agisoft-llc/metashape-scripts

exemple d’utilisation de la console Python dans Metashape, décocher les orientations de caméras :

doc=Metashape.app.document
chunk=doc.chunk
for camera in chunk.cameras:
   camera.reference.rotation_enabled=0

GPS time

GPS time system (recording the absolute number of seconds since the start of the GPS epoch at midnight on January 6th 1980)

Sometimes GPS time is given with an offset (e.g. 1000000000), in this case the 0 value corresponds to 2011/09/14 at 01:46:25 UTC.

Note also that POSIX (Unix) time is given since Jan 1st, 1970, 00:00:00 UTC.

convert to date

835734.87 6250812.50 -3.04 20 0 57560982
835734.88 6250296.85 -8.16 30 0 46351553
835734.88 6250476.48 -6.97 30 0 57559145

example of gps dataset

The gps epoch is in the sixth field, and the values are offset by 1 billion. To convert to date one can use Python.

>>> import datetime
>>> datetime.date.fromtimestamp(1315964785) #posix time of the GPS epoch including the 1 billion offset
datetime.date(2011, 9, 14)
>>> datetime.date.fromtimestamp(1315964782+57560982) # gps time of the dataset + the posix time
datetime.date(2013, 7, 11)

Crop RPC

Using MicMac tools :

example:

mm3d SateLib CropRPC Ori-RPC-d0/GB-Orientation-IMG_PHR1A_P_201902190719128_SEN_3788788101-001.TIF.xml Ori-RPC-d0/GB.* Cropped Org=[3000,4000] Sz=[10500,8500]

parameters:

  1. image to use for the definition of crop zone (Org and Sz correspond to this image)
  2. pattern of orientation files for images to be cropped
  3. name of folder to store cropped RPC
  4. Org : Origin of the box to crop
  5. Sz : Size of the box to crop

Using Proj transformation

PROJ is a generic coordinate transformation software that transforms geospatial coordinates from one coordinate reference system (CRS) to another. It is currently used in many GIS softwares, but can also be used through an API or by command-line tools.

Below some usefull command lines:

Simple conversion from a known coordinate system (CS) to another known CS :

here from UTM33N-WGS84 to WGS84

echo 357000 4676000 | cs2cs +init=epsg:32633 +to +init=epsg:4326

Should return

13d16'2.189"E 42d13'23.141"N 0.000

More documentation https://proj.org/usage/quickstart.html

to add some formatting to output, specify the format with “-f” as a printf format string. For example “-f %.8f” will return the coordinate in decimal degrees with 8 decimals

13.26727481 42.22309481 0.00000000

to input several coordinates at a time, either use a file and pipe with cat

cat coords_utm.txt | cs2cs +init=epsg:32633 +to +init=epsg:4326 -f %.8f Should return the list of converted coordinates
13.26727481 42.22309481 0.00000000
13.54977927 42.05589846 0.00000000

and to output into a file, redirect with > symbol

cat coords_utm.txt | cs2cs +init=epsg:32633 +to +init=epsg:4326 -f %.8f > coords_dd_wgs84.txt

to input manually several coordinates, use EOF or whatever specific characters you like :

cs2cs EPSG:4326 EPSG:32631 <<EOF
45N 2E
EOF

Coordinate system transformation : ITRF, ETRF, WGS84

Clear introduction about Global and local referential (Lantmateriet) : “Positions determined by the GNSS method Precise Point Positioning (PPP) are in the same reference frame as the orbits, i.e. usually a realization of ITRS, e.g. ITRFyy, IGSyy or WGS84, where “yy” represents the year of the realization. The coordinates change with time in the ITRS realizations, because of the plate tectonics. Hence,the determined coordinates are given in the epoch of the observations. For practical applications like mapping and referencing spatial data , a static system/frame, which does not change with time, is desired. For this purpose, ETRS89 has been developed for Europe. ETRS89 coincides with ITRS at epoch 1989.0.”

Another simple reading about dealing with ITRS, ETRS and WGS84 is at Confluence website.

Nota : the transformations made by GIS software from WGS84 to local referential are precise at 1 meter only. For centimeter accuracy, use a geodetic software taking into account the velocities of the ITRF referential relative to the local referential (e.g. ETRF for Europe).

There are many WGS84 realizations. The latest compares with ITRF08 and ITRF14, see table below:

YearRealization (Epoch)For all practical purposes equivalent to:
1987WGS 1984 (ORIG)NAD83 (1986)
1994WGS84 (G730)ITRF91/92
1997WGS84 (G873)ITRF94/96
2002WGS84 (G1150)ITRF00
2012WGS (G1674)ITRF08
2013WGS (G1762)Compares to ITFR08 within 1cm Root Mean Square (RMS) overall
Table of WGS84 compared to ITRF from https://www.e-education.psu.edu/geog862/node/1804

For US, WGS84 (G1762) is equivalent at 1cm to ITRF14. For France also ITRF14 ~ ITRF08 at less than 1cm, the transformation is given here http://itrf.ensg.ign.fr/trans_para.php

For France, the official referential is RGF93, which in its latest version is defined as ETRF2000 (epoch 2009.0) ( https://geodesie.ign.fr/index.php?page=rgf93 ).

Position and velocities of the IGS stations in ITRF2014 at epoch 2010 is given at this adress http://itrf.ensg.eu/ITRF_solutions/2014/doc/ITRF2014_GNSS.SSC.txt

also at https://itrf.ign.fr/ITRF_solutions/2014/doc/ITRF2014_GNSS.SSC.txt

Tools to Convert between referentials

Reference online tool to convert between ITRF and ETRF : http://www.epncb.oma.be/_productsservices/coord_trans/index.php

Coordinate transformation software for France : Circé IGN

Coordinate transformation software : PROJ

Example : converting from ITRFxx (epoch XXXX) to RGF93

We have made a GNSS survey in May 2020 that we want to convert to France official referential in a projection. The GNSS referential will then be ITRF14 (epoch 2020.1), and the French referential will be RGF93 with the associated projection Lambert93. In order to do so, we need to make some referential transformation, and also some conversion between cartesian coordinates, geographic coordinates and projected coordinates.

To transform ITRF14 (epoch 2020.1) to RGF93, use this site and choose ETRF2000 (epoch 2009) as equivalent to RGF93 (see this post), for the velocities you must choose the one of the nearest IGS station (positions and velocities given at ITRF official site or there or Euref site). Then to transform cartesian to geographic coordinates, use Circé IGN software. And finally, use also Circé to convert to projected coordinate system.

More information

GNSS processing with open-source softwares

The main methods of GNSS processing are (1) the classical differential correction with a permanent base station, and (2) the Precise Point Positionning (PPP) method that does not require a base station and uses precises clock and ephemeris data.

The tools used can be softwares or web-services. Among software we can distinguish between commercial softwares (e.g. Trimble Business Center TBC), open-source softwares (e.g. RTKLib), and scientific softwares (Bernese, Gamit/Globk). We will introduce RTKLib and web-services.

RTKLib

RTKLib is the main open-source library and software to process GNSS data. It can do conversion, post-processing, navigation, plotting, and so on.

Note : Several versions of the software exist and they behave somewhat differently. I’ll try to describe them one by one but you’d better check carefully your results compared to another “official” reference in order to validate your process.

Versions of RTKLib :

  • Official software, with actual version 2.4.2. Does not work for me in post-processing as it is not possible to load base station Rinex data.
  • Emlid Fork here. Should be best for conversion of UBX (Emlid raw format) to Rinex. For what I can say, it worked for conversions, for post-processing of the base station, but it was not stable for post-processing of rover.
  • RTKLib_Explorer fork here. Located on the really rich blog about GNSS and RTKLib named RTKLibExplorer, this tool works well and is well documented. The setting of parameters for static processing and obtain a single position is hard (set ON for “Output Single for Sol outage” and set a value for Max Sol Std)

Complementary tools for dealing with POS files from RTKLib and CSV files from ReachView :

PPP web-services

Static GNSS correction with web-service

This processing is done with static GNSS correction using a network of permanent stations.

RGP IGN http://rgp.ign.fr/SERVICES/calcul_online.php , good for France, gives results in many coordinate systems including RGF93-Lambert93

GNSS complentary data and tools

GPS and Glonass ephemeris data, along with atmospheric data at https://kb.igs.org/hc/en-us/articles/115003935351

and also here (search more simple) https://webigs.ign.fr/gdc/fr/product/format#ephem

and also here (search simple too) https://cddis.nasa.gov/Data_and_Derived_Products/GNSS/orbit_products.html

IGS Antenna calibration file (does not contain Emlid Reach RS2) : igs14.atx

List of IGS stations : here

GPS Calendar to get the GPS week : here or at IGN site

Julian calendar to get GPS day : here or at IGN site

Offset between GPST (GPS Time) and UTC : here

More information