apt Module

Apt command line installer and package manager for Osgeo4W.

Examples

Typical daily use:

apt update                   (fetch up-to-date setup.ini)
apt install gdal gdal-python (install packages "gdal" and "gdal-python", and dependencies)
apt new                      (show possible upgrades)
apt list                     (show installed packages)
apt available                (show installation candidates)
apt remove xxx yyy           (uninstall packages xxx and yyy)

Notes

Apt strives to match Osgeo4wSetup.exe’s results as closely as possible, and uses the same configuration and install manifest files. A prime directive is that user’s should never be put in a position where they feel the need to choose between the tools and not go back.

That’s the aspiration. There’s no guarantee it’s been achieved.

At the moment apt can only install the 32bit Osgeo4W packages.

References

[1]	Based on cyg-apt - “Cygwin installer to keep cygwin root up to date” (c) 2002–2003 Jan Nieuwenhuizen <janneke@gnu.org> License: GNU GPL Modified by Matt.Wilkie@gov.yk.ca for OSGeo4W, beginning July 2008

apt.available(dummy)

Show packages available on the mirror.

Display packages available on the mirror, with installed packages marked *. Specify an alternate mirror with --mirror=...

Parameters:dummy (str) – Parameter is not used at present. Returns:Package names (without install mark). Return type:list

apt.ball(packages)

Print full local path name of package archive

C:> apt ball shell

shell = d:/temp/o4w-cache/setup/http%3a%2f%2fdownload.osgeo.org%2fosgeo4w%2f/x86

/release/shell/shell-1.0.0-13.tar.bz2

FIXME: This should either return a list of archive filenames, or there should be a get_ball(p) which returns 1 filename, or we should rip out all this repetitive code spread across multiple functions, for the purpose of allowing multiple package input. We need a handler for this instead.

apt.check_env(o4w='')

Verify we’re running in an Osgeo4W-ready shell

apt.check_setup(installed_db, setup_ini)

Look to see if the installed packages db and setup.ini are available

apt.datetime_to_unixtime(dt, epoch=datetime.datetime(1970, 1, 1, 0, 0))

Convert a datetime object to unix UTC time (seconds since beginning).

Adapted from http://stackoverflow.com/questions/8777753/converting-datetime-date-to-utc-timestamp-in-python/

It wants from __future__ import division, but that caused issues in other functions, automatically coverting what used to produce integers into floats (e.g. “50/2”). It seems to be safe to not use it, but leaving this note just in case…

apt.do_download(packagename)

Download package from mirror and save in local cache folder.

Overwrites existing cached version if md5 sum doesn’t match expected from setup.ini.

Returns path orchive.bz2 on success (file downloaded, or file with correct md5 is present), and http status code if fails.

apt.do_install(packagename)

Unpack the package in appropriate locations, write file list to installed manifest, run postinstall confguration.

apt.do_run_preremove(root, packagename)

Run the etc/preremove batch files for this package

apt.do_uninstall(packagename)

For package X: delete installed files & remove from manifest, remove from installed.db

apt.dodo_download(url, dstFile)

Dumbest name for abstracting downloading a file to disk with requests module and progress reporting

Returns path orchive.bz2 on success, http status code if fails.

apt.down_stat(downloaded_size, total_size)

Report download progress in bar, percent, and bytes.

Each bar stroke ‘=’ is approximately 2%

Adapted from

http://stackoverflow.com/questions/51212/how-to-write-a-download-progress-indicator-in-python http://stackoverflow.com/questions/15644964/python-progress-bar-and-downloads

apt.download(packages)

Download the package(s) from mirror and save in local cache folder:

C:> apt download shell gdal {…etc}

shell = d:/temp/o4w-cache/setup/http%3a%2f%2fdownload.osgeo.org%2fosgeo4w%2f/x86/release/shell/shell-1.0.0-13.tar.bz2 remote: c38f03d2b7160f891fc36ec776ca4685 shell-1.0.0-13.tar.bz2 local: c38f03d2b7160f891fc36ec776ca4685 shell-1.0.0-13.tar.bz2

gdal = d:/temp/o4w-cache/setup/http%3a%2f%2fdownload.osgeo.org%2fosgeo4w%2f/x86/release/gdal/gdal-1.11.1-4.tar.bz2 remote: 3b60f036f0d29c401d0927a9ae000f0c gdal-1.11.1-4.tar.bz2 local: 3b60f036f0d29c401d0927a9ae000f0c gdal-1.11.1-4.tar.bz2

Use apt available to see what is on the mirror for downloading.

apt.exceptionHandler(exception_type, exception, traceback, debug_hook=<built-in function excepthook>)

Print user friendly error messages normally, full traceback if DEBUG on. Adapted from http://stackoverflow.com/questions/27674602/hide-traceback-unless-a-debug-flag-is-set

apt.find(patterns)

Search installed packages for filenames matching the specified text string.

apt.get_all_dependencies(packages, nested_deps, parent=None)

Recursive lookup for required packages in order of dependence. Returns an ordered list.

apt.get_arch(bits='')

Determine CPU architecture to use (X86, X86_64) from –arch parameter. Allows –arch 32 | 64 as well as longer –arch x86 | x86_64

apt.get_cache_dir()

Return path to use for saving downloads.

Precedence order:

• command line option (-c, –cache)
• last used cache (read from setup.rc)
• Public Downloads folder
• Osgeo default (%osgeo4w_root%/var/…)

apt.get_config(fname)

Open /etc/setup/fname and return contents, e.g. /etc/setup/last-cache

apt.get_filelist(packagename)

Retrieve list of files installed for package X from manifest (/etc/setup/package.lst.gz)

apt.get_info(packagename)

Retrieve details for package X.

Returns dict of information for the package from dict created by parse_setup_ini() (category, version, archive name, etc.)

apt.get_installed()

Get list of installed packages from ./etc/setup/installed.db.

Returns nested dictionary (empty when installed.db doesn’t exist): {status_int : {pkg_name : archive_name}}

I don’t know significance of the nesting or leading zero. It appears to be extraneous? The db is just a straight name:tarball lookup table. In write_installed() the “status” is hard coded as 0 for all packages.

apt.get_installed_version(packagename)

Derive version number from archive filename in ‘installed’ dict.

apt.get_menu_links(bat)

Parse postinstall batch file for menu and desktop links.

Relies on shlex module which splits on spaces, yet preserves spaces within quotes (http://stackoverflow.com/questions/79968)

apt.get_missing(packagename)

For package, identify any requirements (dependencies) that are not installed.

Returns a dictionary of {packagname: [‘missing_1’,’missing_2’,’…’]}

apt.get_new()

Return list of mirror packages of newer versions than those installed.

apt.get_requires(packagename)

identify dependencies of package [deprecated]

use get_all_dependencies() for recursive dependencies list and get_info(p)[‘requires’] for just one level

apt.get_setup_arch(setup_ini)

Return CPU architecture used in setup.ini

apt.get_special_folder(intFolder)

Fetch paths of Windows special folders: Program Files, Desktop, Startmenu, etc.

Written by Luke Pinner, 2010. Code is public domain, do with it what you will… todo: look at replacing with WinShell module by Tim Golden, http://winshell.readthedocs.org/en/latest/special-folders.html

apt.get_zipfile(packagename)

Return full path name of locally downloaded package archive.

apt.hashcheck(package)

Check if the md5 hash for “package” in local cache matches mirror

> apt hashcheck shell

Returns: True or False for md5 match status

None when cache file not found

If passed a list it only processes the first item.

apt.help(*args)

Show help for COMMAND

apt.info(packages)

info - report name, version, category, etc. about the package(s)

B:> apt info shell

name : shell version : 1.0.0-13 sdesc : “OSGeo4W Command Shell” ldesc : “Menu and Desktop icon launch OSGeo4W command shell” category : Commandline_Utilities requires : msvcrt setup zip_path : x86/release/shell/shell-1.0.0-13.tar.bz2 zip_size : 3763 md5 : c38f03d2b7160f891fc36ec776ca4685 local_zip: d:/temp/o4w-cache/setup/http%3…/shell-1.0.0-13.tar.bz2 installed: True install_v: 1.0.0-11

Notes

• “local_zip” is best guess based on current mirror. (We don’t record which mirror was in use at the time of package install.)
• “version” is from setup.ini, what is available on the mirror server
• “install_v” is the version currently installed

apt.install(packages, force=False)

Download and install packages, including dependencies

C:> apt install shell gdal

apt.list_installed(dummy)

List installed packages

apt.listfiles(packages)

List files installed with package X. Multiple packages can be specified.

C:> apt listfiles shell gdal

—– shell —– OSGeo4W.bat OSGeo4W.ico bin …etc

—– gdal —– bin bin/gdal111.dll bin/gdaladdo.exe …etc

apt.missing(dummy)

Display missing dependencies for all installed packages.

dummy parameter is ignored

apt.new(dummy)

List available upgrades to currently installed packages

apt.parse_setup_ini(fname)

Parse setup.ini into package name, description, version, dependencies, etc.

Parameters:fname – full path to setup.ini Returns:{Distribution {Program_name{[‘category’, ‘source’, ‘ldesc’, ‘version’, ‘install’, ‘sdesc’, ‘requires’]}}}

{curr {

‘gdal’ {

‘name’: ‘gdal’, ‘version’: ‘1.11.1-4’, ‘category’: ‘Libs Commandline_Utilities’, etc… }

}}

Return type:A nested dictionary

apt.parse_setuprc(fname)

Parse setup.rc config file into a dictionary.

We assume any line beginning with a tab is a value, and all others are dict keys. Consequently this will return a bad dict if there are extra lines starting with tabs.

Example C:OSGeo4Wetcsetupsetup.rc:

mirrors-lst

http://download.osgeo.org/osgeo4w/;OSGeo;USA;California

window-placement

44,0,0,0,0,0,0,0,1,0,0,0,255,255,255,255,255,255,255,255…

last-mode

Advanced

last-mirror

http://download.osgeo.org/osgeo4w/

net-method

Direct

last-cache

C:UsersMattDownloads

last-menu-name

OSGeo4W_default

And result:

last-cache: C:UsersMattDownloads last-mirror: http://download.osgeo.org/osgeo4w/ mirrors-lst: http://download.osgeo.org/osgeo4w/;OSGeo;USA;Cal… window-placement: 44,0,0,0,0,0,0,0,1,0,0,0,255,255,255,255… last-mode: Advanced last-menu-name: OSGeo4W_default net-method: Direct

apt.post_install(packagename)

Run postinstall batch files and update package manifest

to catch those files not included in the package archive. (manifest = etc/setup/pkg-foo.lst.gz)

adapted from “17.1.3.3 Replacing os.system()” http://www.python.org/doc/2.5.2/lib/node536.html

apt.remove(packages)

Uninstall listed packages

apt.requires(packages)

What packages does X rely on?

Returns dictionary of package names and dependencies. Reports sub-dependencies, but they aren’t in the dict (yet).

apt.search(pattern)

Search available packages list and descriptions for X

Returns list of package names

apt.set_extended_info(d)

set extended information into package-info-dictionary, as used by get_info() or parse_setup_ini()

We take compound values in single keys and explode them into their own keys.

{‘install’: ‘x86/release/gdal/gdal-1.11.1-4.tar.bz2 5430991 3b60f036f0d29c401d0927a9ae000f0c’}

becomes:

{‘zip_path’: ‘x86/release/gdal/gdal-1.11.1-4.tar.bz2’} {‘zip_size’:‘5430991’} {‘md5’:‘3b60f036f0d29c401d0927a9ae000f0c’}

apt.setup(target)

Create skeleton Osgeo4W folders and setup database environment

apt.split_ball(filename)

Parse package archive name into a) package name and b) version numbers tuple (to feed into version_to_string)

mc-4.6.0a-20030721-12.tar.bz2

mc –> package name 4.6.0a-20030721 –> upstream application version 12 –> package version

python-numpy-2.7-1.5.1-1.tar.bz2

python-numpy –> package name 2.7-1.5.1 –> upstream application version 1 –> package version

Returns:(‘mc’, (4, 6, 0a, 20030721, 12)) (‘python-numpy’, (2, 7, 1, 5, 1, 1))

apt.uniq(alist)

Returns a list with unique items (removes duplicates), without losing item order. From @jamylak, http://stackoverflow.com/a/17016257/14420

apt.unique(L)

Remove duplicates and empty items from a list

apt.update()

Fetch updated package list from mirror.

apt update

Specify mirror (web server, windows file share, local disk):

apt –mirror=http://example.com/… update apt –mirror=file:////server/share/… update apt –mirror=file://D:/downloads/cache/… update

apt.upgrade(packages)

Upgrade named packages.

apt upgrade all apt upgrade gdal-filegdb qgis-grass-plugin

apt.url(packages)

Print remote package archive path, relative to mirror root

apt.url_time_to_datetime(s)

Convert “last-modified” string time from a web server header to a python datetime object.

Assumes the string looks like “Fri, 27 Mar 2015 08:05:42 GMT”. There is no attempt to use locale or similar, so the function is’nt very robust.

apt.version(packages)

Report installed version of X

apt.write_installed()

Record installed packages in install.db

apt.write_setuprc(setuprc, fname='setup.rc')

Write the setuprc dictionary to file, in osgeo4w-setup.exe format.

Dict entries with empty values are left out.

Incoming dict:

last-mode: None last-mirror: http://download.osgeo.org/osgeo4w/ net-method: None last-cache: C:UsersMattDownloads last-menu-name: OSGeo4W_default

Out etc/setup/setup.rc:

last-mirror

http://download.osgeo.org/osgeo4w/

last-cache

C:UsersMattDownloads

last-menu-name

OSGeo4W_default

class apt.xAttrDict(*args, **kwargs)

Access a dictionary by attributes, like using javascript dotted notation.

dict.mykey <— same as —> dict[‘mykey’]

From http://stackoverflow.com/questions/4984647/accessing-dict-keys-like-an-attribute-in-python

apt.xrequires(packages)

https://github.com/maphew/apt/issues/32