Package Reference

defs - Basic definitions for adapya

The module defs.py in the adapya.base package defines basic functionality for the adapya packages

• Abuf class (read/write buffer) provides byte buffers with read and write access funtions

• Logging

• dummymutex for synchronizing printing and logging

adapya.base.defs.Abuf(init, encoding='utf-8', errors='strict')

Create a read/write buffer similar to a bytearray. This factory function returns a Cbuf object.

param init

a buffer size > 0 or an initial nonempty string or unicode string which determines the size of the buffer.

param encoding

standard encoding encode string to binary string, default is ‘utf-8’

Some Abuf use examples (Python2):

>>> a=Abuf(10)
>>> a.write('abc')
>>> a.write('DEFG')
>>> a.value
b'abcDEFG'
>>> a.raw
b'abcDEFG\x00\x00\x00'
>>> a.tell()
7
>>> a.seek(0)
>>> a.read(3)
b'abc'
>>> a.read_text(4)
'DEFG'
>>> a[0]
b'a'
>>> a[0:3]
b'abc'
>>> a[3:]=b'1234567'
>>> a.value
b'abc1234567'

Some examples (Python3):

>>> e=Abuf(10,encoding='cp037')
>>> e.write('ABC')
>>> e.write(b'123')
>>> e.value
b'\xc1\xc2\xc3123'
>>> e.raw
b'\xc1\xc2\xc3123\x00\x00\x00\x00'
>>> e.seek(0)
>>> e.read_text(3)
'ABC'
>>> e.read(3)
b'123'
>>> e[0]
b'Á'
>>> e[0:3]
b'ÁÂÃ'

exception adapya.base.defs.ProgrammingError

adapya.base.defs.evalb(s)

Evaluates string for single byte escape sequences

Returns

byte string

>>> evalb('ABCäÿ')
b'ABCäÿ'

Note

In PY3 no unicode escapes can be used, use evals() for that purpose.

adapya.base.defs.evals(s)

Evaluates string for unicode escape sequences

Returns

string

>>> evals('ABCäÿ')
'ABCäÿ'

>>> evals('ABCäÿ')
'ABCäÿ'

Note

In PY3 no escaped single bytes can be defined in strings - they are mapped to unicode characters.

use evalb() for that purpose.

adapya.base.defs.log(logparm=None)

set adapya.base interface log options

conv - Text conversion routines

The module conv.py contains several conversion routines for ASCII/EBCDIC conversion and byte swapping.

Character set is Latin1 per default with cp37 as EBCDIC and cp819 (ISO-8859-1) as extended ASCII code pages.

The following conversions are defined:

  37 <=> 819
1047 <=> 819
1141 <=> 1252

adapya.base.conv.asc2ebc(buf, start, stop, senco=819, tenco=37)

convert ASCII bytes in buffer to EBCDIC

Parameters

• buf – buffer

• start – start offset in buf

• stop – ending offset in buf

• senco –

  source ASCII encoding. Supported are:

  819 (Latin1 ISO-8859-1) or 1252 (Windows Latin1)

• tenco –

  target EBCDIC encoding. Supported are:

  37 (US EBCDIC Latin1) or 1141 (US EBCDIC with Euro)

adapya.base.conv.ebc2asc(buf, start, stop, senco=37, tenco=819)

convert EBCDIC bytes in buffer to ASCII

Parameters

• buf – buffer

• start – start offset in buf

• stop – ending offset in buf

• senco –

  source EBCDIC encoding. Supported code pages

  37 (US EBCDIC Latin1) or 1141 (US EBCDIC with Euro)

• tenco –

  target ASCII encoding. Supported code

  819 (Latin1 ISO-8859-1) or 1252 (Windows Latin1)

adapya.base.conv.str2asc(istr, senco=37, tenco=819)

Translate bytes in bytes, bytearray or string to target encoding and return bytes string

adapya.base.conv.str2ebc(istr, senco=819, tenco=37)

Translate bytes in bytes, bytearray or string to target encoding

>>> str2ebc(b'ABC')
b'ÁÂÃ'

>>> abc=bytearray('ABC',encoding='Latin1')

>>> str2ebc(abc)
bytearray(b'ÁÂÃ')

>>> str2ebc('ABC')
b'ÁÂÃ'

adapya.base.conv.str2uni(b)

Convert bytes string to unicode string. The length of the input string s must be even otherwise a ValueError exception is raised.

adapya.base.conv.swap(s)

swap string, bytes or bytesarray

adapya.base.conv.uni2str(u)

convert unicode string to bytes string

Parameters

u – string

Returns

bytes string

datamap - Map Memory Structures to Class Attributes

The datamap.py module defines the Datamap class which allows to map record or memory structures to class attributes

adapya.base.datamap.Bytes(name, size, **options)

Byte string (binary)

class adapya.base.datamap.Datamap(dmname, *fieldlist, **kw)

Datamap maps attributes to fields located in buffer+offset.

This is similar to a struct in C and involves un-/packing to/from Python objects.

It is used to handle data structures external to Python.

Internally, attributes are defined as tuples per field list parameter and stored as keys in the dict ‘keydict’ that contains tuples of

• data_type,

• slice_start,

• slice_length,

• read/write ability

• function or method to convert to readable string

The sequence of fields is maintained in the the list ‘keylist’

Attribute tuples are generated with a list of data type specific functions. Each defines the attribute name and other field options

Examples:

>>> from adapya.base.defs import Abuf
>>> from adapya.base.datamap import Datamap, String, Int2
>>> g = Datamap( 'mymap',String('foo', 6),Int2('bar') )
>>> g.buffer=Abuf(8)

>>> g.bar=255
>>> g.foo='abcdef'

>>> print( g.foo)         # attribute access
abcdef

>>> dump(g.buffer)      # contents of buffer
Buffer
 0000 61626364 6566FF00                   abcdef..         /ÂÄÀÁÃ..

>>> print( '%(foo)s and %(bar)d' % g)        # dictionary access
abcdef and 255

The following keywords are supported:

• buffer sets the buffer on which the datamap is mapped

• offset defines the offset in the buffer where the datamap starts

• encoding sets the encoding to convert to/from Unicode (default ‘Latin1’)

• ebcdic if True: Alpha strings are in EBCDIC (encoding will be set to ‘cp037’ if not encoding is not specified)

• byteOrder define byte order for whole datamap (NETWORKBO or NATIVEBO)

• occurs multiple occurrence of datamap, e.g. PE group is a fixed number or an occurrence function, for example

  occurs=lambda:m1.occ - is a reference the occurrence count

• dmlen

• varies indicates variable field positions and lengths

• supermap enclosing datamap which provides defaults for buffer, offset etc.

dprint(indent=0, proff=0, selectfields=(), skipnull=0, title='')

Print detail lines with all attributes

Parameters

proff – 1 = print offset when walking through buffers

Parm selectfields

display only fields listed (optional)

Parm skipnull

1 = do not display empty fields

Parm title

title to display else datamap name

genfb()

Generate Format buffer from datamap with fields defining Adabas field name with options fn

datamap instance must have field name fn added to the options dict of each field

Returns

string with format buffer contents

Note: currently limited to normal fields - no PE/MU support

>>> from adapya.base.datamap import Datamap,Int2,String
>>> g = Datamap('mymap',String('foo',6,fn='AA'),Int2('bar',fn='AB',occurs=3))
>>> g.genfb()
'AA,6,A,AB,2,F.'

getfndef(name)

Get Adabas field definition from datamap

Returns

tuple Adabas (field name, length, format) or None if name does not exist or field name not found

>>> g = Datamap( 'mymap',String('foo',6,fn='AA'),Int2('bar',fn='AB') )
>>> g.getfndef('foo')
('AA', 6, 'A')

getpossiz(index, opt, fieldpos)

Determine position and size in variable field or occ. count before MU/PE field ( opt & T_MUPE)

called from prepare()

getsize()

Returns

size of datamap

items(selectfields=())

Return list of name value pairs on the fields in datamap

Restriction: no composite fields

todo

check if byte arrays/fields can be used when handling bytes

>>> from adapya.base.datamap import Datamap, String, Int2
>>> g = Datamap( 'mymap',String('foo', 6),Int2('bar') )
>>> from adapya.base.defs import Abuf
>>> g.buffer=Abuf(8)
>>> g.bar=255
>>> g.foo='abcdef'
>>> g.items()
[('foo', 'abcdef'), ('bar', 255)]

lprint(header=0, indent=0, proff=0, selectfields=(), col1='')

Print line with all attributes in one line with

Parameters

• header – 1: print only header

• indent – Columns to indent (default = 0)

• proff – 1: print offset when walking through buffers

• selectfields – display only fields listed (optional)

• col1 – prefix text as first column (use to add index numbers to line) Header and detail line should pass same string length

prepare()

Prepare datamap for field access.

This function must be called if datamap contains variable fields or variable number of occurences.

It sets the exact field position [1] and size [2] in the field definition list so that field access can use it.

For variable length fields position/size excludes the length part initsize - stores the initial size (=0 for variable fields).

reset()

Reset attributes/key values of a datamap to default values if buffer variable is set:

>> adac.cb.reset()

rippledown(fun, parm)

apply function to lower datamaps no data is returned

setEbcdic(yesno)

set String/Char data to EBDIC in datamap

This method also overrides any encoding with ‘cp037’ for EBCDIC or ‘Latin_1’ for not EBCDIC.

Parameters

yesno – 0 to switch to EBCDIC or 1 to switch to ASCII

setfsize(key, newsize)

set new fieldsize Note: offsets of the following fields are not adjusted!

update(*kviter, **kw)

Update or set the attributes/keys of a datamap to the values in the list given as list or iterable of (key, value) pairs or as list of keyword=value itmes

Parameters

• kviter – list or iterable of ‘key’,value pairs

• kw – list of key=value

Example:

>> empl.update(name='Bell',first_name='John')

exception adapya.base.datamap.DatamapError(value, dmap)

Error Class for Datamap Exceptions

Attr string

Adabas response string

Attr dmap

Datamap instance where the error occurred

Usage example:

try:
    raise DatamapError('error text', dmap)
except DatamapError as e:
    print( 'DatamapError', e.value, e.__class__)
    dump(e.dmap)

adapya.base.datamap.Filler(name, size)

Byte filler

class adapya.base.datamap.Multiple(supermap, superkey, occurs=1, submap=None)

Initialize with Multiple(supermap, key, occurs [, submap]) if ‘occurs’ is a function it will be called in prepare() to determine the actual occurrences before field access

This class is used internally when defining fields with occurs>0

adapya.base.datamap.Periodic(dmap, occurs=0)

Define periodic group with a datamap dmap on the fields of the group keywords for options:

Parameters

occurs –

number of repetitions of group, parameter may be a function.

If function returns -1 or -2: byte position(s) before the field contain the actual count. For example:

occurs=lambda:-2

adapya.base.datamap.Unicode(name, size, **options)

size in unicode characters * 2 = size in bytes

adapya.base.datamap.bit_str(b, istrList)

Produce readable print of integer or single byte input

Parameters

• b – byte or integer

• istrList –

  list of tuples containing the bits and the corresponding string.

  Optionally a third element of a tuple may be a string that is taken as default when the bits are all not set.

Examples:

>>> from adapya.base.datamap import bit_str
>>> print( bit_str(b'\x03',[(1,'one'),(2,'two')]))
one,two

>>> print( bit_str(b'\x03',[(3,'three'),(1,'one'),(2,'two')]))
three

>>> print( bit_str(b'\x06',[(3,'three'),(1,'one'),(2,'two')]))
two,X'04'

>>> bit_str(b'\x86',[(2,'two'),(0x80,'negative','positive')])
"two,negative,X'04'"

>>> bit_str(b'\x00',[(2,'two'),(0x80,'negative','positive')])
'positive'

adapya.base.datamap.dunpack(dmap, key, indx=0, possiz=None)

datamap unpack

Parameters

• index – if index is > 0: indexed access with constant field size>0 if possiz not given

• possiz – list of (pos, size) tuple indexed by index

adapya.base.datamap.field(name, fmt, size, **options)

Defines a field element.

A field element is used in the list of fields in a datamap class. It defines mainly name, format and size.

Valid keywords for options i.e. keyword = value list:

Parameters

• opt –

  define extra formatting e.g. (T_STCK, T_HEX) - see also below on variable length strings

  To define fields in other data architecture:

  • T_EBCDIC field data is in EBCDIC (String, Packed, Unpacked, Char)

  • T_NWBO field data is big-endian or high-order-byte-first rather than native byte-order

  To define this for all fields in a datamap use the setting on the datamap level:

  Datamap(…, ebcdic=1, byteorder=NETWORKBO)

  To define variable length strings:

  opt in (T_VAR0, T_VAR1, T_VAR2, T_VAR4)

  for AAL length reference, 1, 2, 4 byte length prefix

  To cumulate options join them with bitwise-or ‘|’

• fn – define Adabas field name. This is used in genfb() and getfndef() functions to generate the format buffer related to the datamap.

• pos – define new offset from start of datamap

• repos – reposition relative from current position

• caption – field text for dprint() or lprint()

• ppfunc – formatting function for dprint()

• colsize – column size for lprint()

• dt – ‘DATETIME’ (= Adabas datetime editmask) is used to map to/from Python datetime object (currently only DATETIME)

• sizefunc – reference to size function if T_VAR0 (AAL)

adapya.base.datamap.flag_str(flag, flist)

This is similar to bit_str but only lists the elements for which a definition exists.

Parameters

• flag – integer

• flist – list of (integer, string) tuples

Example:

>>> flag_str(3, ((1,'one'),(2,'two'),(4,'four')))
'one,two'

adapya.base.datamap.flag_strc(flag, flist)

Similar to flag_str but compressed i.e. without commas each flag bit is represented by a string or a ‘.’

Parameters

• flag – integer

• flist – list of (integer, string) tuples

Example:

>>> flag_strc(5, ((1,'O'),(2,'T'),(4,'F')))
'O.F'

adapya.base.datamap.fndef2datamap(name, fndef)

Return a Fieldmap class from a fndef list :param fndef: list of fndef tuples :param name: class name to be returned

adapya.base.datamap.fpack(value, format, length, byteorder=None, ebcdic=0)

Pack number to byte string in one of the following number representations/formats supported by Adabas:

Unpacked, Packed, Binary, Fixpoint, unsigned integer ‘u’

Parameters

• value – number (int/long)

• format – ‘U’, ‘P’, ‘B’, ‘F’, ‘u’

• length – allowed are for U:1-30, P:1-15, B:1-126, F:1,2,4,8 u:1,2,4,8

Returns

string in PY2 and bytes in PY3

Example:

>>> from adapya.base.datamap import fpack
>>> fpack(234,'P',2) == b'#L'
True

adapya.base.datamap.funpack(bstring, format, byteorder=None, ebcdic=0)

Unpacks number from byte string in one of the following number representations/formats supported by Adabas:

Unpacked, Packed, Binary, Fixpoint

Parameters

• value – Byte string

• format – ‘U’, ‘P’, ‘B’, and ‘F’ plus ‘u’ unsigned

• byteorder – NETWORKBO or NATIVEBO (defined in datamap)

Returns

number (int/long) for ‘U’, ‘P’, ‘F’ and ‘u’ or byte string for ‘B’

Example:

>>> from adapya.base.datamap import funpack
>>> funpack(b'#M','P')
-234

adapya.base.datamap.list_str(i, istrList)

Produce readable print from list

Parameters

• i – position in list

• istrList – List of strings

adapya.base.datamap.list_stri(i, istrList)

Produce readable print from list

Parameters

• i – position in list

• istrList – List of strings

adapya.base.datamap.prd2dm(prdformat, length=0)

get Datamap field code from Predict format

adapya.base.datamap.setEbcdic(yesno)

set EBCDIC data in map :param yesno: True or False

adapya.base.datamap.str_str(s, strdict)

Return a readable string from the value if it exists otherwise the hex value for integers or blank

Parameters

• s – value (could be string or int: key for dict)

• strdict – dict containing the interpretation

Examples:

>>> str_str('quark', {'quark':'one',2:'two',4:'four'})
'one'
>>> str_str(3, {1:'one',2:'two',4:'four'})
"X'03'"

dtconv - Date and Time conversion functions

The module adapya.base.dtconv contains various routines to convert datetime values between various formats.

For Gregorian to Julian date conversions and vice versa see

http://en.wikipedia.org/wiki/Julian_day#Calculation

exception adapya.base.dtconv.InvalidDateException

adapya.base.dtconv.checkdate(year, month, day)

Check Gregorian date given as year, month, day

Raises

InvalidDateException – if invalid input values are given otherwise returns silently

Invalid date raises exception:

>>> checkdate(2008,12,32)   
Traceback (most recent call last):
InvalidDateException: day 32 not in range 1..31

Valid dates go trough quiet:

>>> checkdate(2008,12,31)

adapya.base.dtconv.clock_time(resync=300)

Parameters

resync – interval for resyncing with time()

Returns

high resolution time

adapya.base.dtconv.date2natdate(year, month, day)

Convert a gregorian date to a natdate integer

No checking is done for valid dates. Natural does not allow dates earlier than 1582-01-01

>>> date2natdate(1,1,1)
365
>>> date2natdate(0,1,1)
-1
>>> date2natdate(0,1,2)
0
>>> date2natdate(1900,1,1)
693960
>>> date2natdate(1582,1,1)
577813
>>> date2natdate(1970,1,1)
719527
>>> date2natdate(2000,1,1)
730484
>>> date2natdate(2008,12,31)
733771
>>> date2natdate(9999,12,31)
3652423

adapya.base.dtconv.dt2str(*dt)

make a datetime string from a datetime tuple

>>> dt2str( 2010,4,30, 11,55,13)
'20100430115513'

adapya.base.dtconv.dt2strf(*dt)

>>> dt2strf( 2010,4,30, 11, 55, 13)
'2010-04-30 11:55:13'

>>> d1=(2010,1,2,11,44,55)
>>> dt2strf( *d1)
'2010-01-02 11:44:55'

adapya.base.dtconv.greg2jdn(year, month, day)

Calculate the Julian day number for a proleptic gregorian date:

November 24, 4714 BC is Julian day 0 at noon

adapya.base.dtconv.gx2utc(halfhours, microsecs)

Convert intermediate timestamp of two 32 bit integers to utc form:

year,month,day,hour,minute,second,microsecond

Parameters

• halfhours – number of halfhours since the epoch 0000-01-02

• microsecs – number of microseconds within half hour

>>> gx2utc(0,0)                     # 0000-01-02 00:00:00.000000
(0, 1, 2, 0, 0, 0, 0)

>>> gx2utc(175316351,1799999999)    # 9999-12-31 23:59:59.999999
(9999, 12, 31, 23, 59, 59, 999999)

>>> gx2utc(34537296,0)              # 1970-01-01 00:00:00.000000
(1970, 1, 1, 0, 0, 0, 0)

>>> gx2utc(35221034,1259123456)     # UTC2008SAMPLE
(2008, 12, 31, 13, 20, 59, 123456)

adapya.base.dtconv.jdn2greg(jdn)

Calculate gregorian year, month, day from julian day number.

astronomical year numbering year 0 is 1 B.C., -1 is 2 B.C. etc.

Even though the first day in the Julian calendar correctly starts at noon rather than on midnight here we assume it to start at midnight (12 hours earlier)

adapya.base.dtconv.jdn2jul(jdn)

Calculate julian year, month, day from julian day number

adapya.base.dtconv.jul2jdn(year, month, day)

Calculate Julian day number from julian date

• Earliest supported date is March 1, -4800

• 1 BC is year 0, 2 BC is -1 etc. (astronomical year numbering)

• Julian year starts at 1. March

adapya.base.dtconv.micro2nattime(mic)

Convert microseconds since 0001-01-01 to NATTIME Value in 10th of second precision

adapya.base.dtconv.micro2utc(microsecs)

Convert seconds since epoch value to list composed of:

year,month,day,hour,minute,second,microsecond

Parameters

microsec – number of microseconds since the epoch 0001-01-01

>>> micro2utc(0)                    # 0001-01-01 00:00:00.000000
(1, 1, 1, 0, 0, 0, 0)

>>> micro2utc(315537897599999999)  # 9999-12-31 23:59:59.999999
(9999, 12, 31, 23, 59, 59, 999999)

>>> micro2utc(62135596800000000)   # 1970-01-01 00:00:00.000000
(1970, 1, 1, 0, 0, 0, 0)

>>> micro2utc(63366326459123456)   # UTC2008SAMPLE
(2008, 12, 31, 13, 20, 59, 123456)

adapya.base.dtconv.natdate2nattime(nd)

convert NATDATE counting days since Jan. 2, 0000 to NATTIME Value in 10th of second precision

adapya.base.dtconv.nattime2micro(nt)

convert NATTIME Value in 10th of second precision to microseconds since 1.1.1

adapya.base.dtconv.nattime2natdate(nt)

Convert NATTIME Value in 10th of second precision to NATDATE couning days since Jan. 2, 0000

adapya.base.dtconv.nattime2utc(nt)

convert NATTIME Value in 10th of second precision to DATETIME value

adapya.base.dtconv.sec2interval(seconds)

Convert seconds to tuple of day, hour, minute and second

Parameters

seconds – integer

Returns

tuple day, hour, minute, second

adapya.base.dtconv.sec2utc(seconds)

Convert seconds since epoch value to tuple composed of

year,month,day,hour,minute,second

Parameters

seconds – number of seconds since the epoch 0001-01-01 if a float value is given: fraction of seconds yield microseconds

adapya.base.dtconv.str2dt(ds)

make a datetime tuple from a datetime string or number

>>> str2dt( '20160229115513' )
(2016, 2, 29, 11, 55, 13)
>>> str2dt( b'20160301235513' )
(2016, 3, 1, 23, 55, 13)

adapya.base.dtconv.testgreg2jdn()

Conversion of seconds to gregorian date

adapya.base.dtconv.ts2strf(*ts)

>>> ts2strf( 2010,4,30, 11, 55, 13, 999999)
'2010-04-30 11:55:13.999999'

adapya.base.dtconv.unix2utc(seconds)

same as sec2utc() only with epoch 1970-01-01

adapya.base.dtconv.utc2gx(*secmsec)

Convert list composed of year,month,day,hour,minute,second,microsecond to gx format list of two 32 bit integers (halfhours, microsecs) since epoch 0000-01-02

Parameters

secmsec – list of year,month,day,hour,minute,second,microsecond

Returns

(halfhours, microsecs) since epoch 0000-01-02

adapya.base.dtconv.utc2micro(*secmsec)

Convert list composed of year,month,day,hour,minute,second,microsecond

Parameters

secmsec – list of year,month,day,hour,minute,second,microsecond

Returns

number of microseconds since the epoch 0001-01-01 00:00:00.000000

XTIMESTAMP(epoch 0001)

>>> utc2micro(*UTCMIN)           # 0001-01-01 00:00:00.000000
0
>>> '%d' % utc2micro(*UTCMAX)    # 9999-12-31 23:59:59.999999
'315537897599999999'
>>> '%d' % utc2micro(*UTC1970)   # 1970-01-01 00:00:00.000000
'62135596800000000'
>>> '%d' % utc2micro(*UTC2008SAMPLE)    # 2008-12-31 13:20:59.123456
'63366326459123456'

XTIMESTAMP(epoch 1970)

>>> '%d' % (utc2micro(*UTCMIN)-utc2micro(*UTC1970))
'-62135596800000000'
>>> '%d' % (utc2micro(*UTCMAX)-utc2micro(*UTC1970))
'253402300799999999'
>>> '%d' % (utc2micro(*UTC1970)-utc2micro(*UTC1970))
'0'
>>> '%d' % (utc2micro(*UTC2008SAMPLE)-utc2micro(*UTC1970))
'1230729659123456'

adapya.base.dtconv.utc2nattime(*secmsec)

convert utc timestamp since 0001-01-01 to NATTIME Value in 10th of second precision

adapya.base.dtconv.utc2sec(*utc)

Convert list composed of year,month,day,hour,minute,second [,microsec]

Parameters

utc – list of year,month,day,hour,minute,second

Returns

number of seconds since the epoch 0001-01-01

adapya.base.dtconv.utc2unix(*utc)

same as sec2utc() only with epoch 1970-01-01

adapya.base.dtconv.utc2xts(*secmsec)

Convert list composed of year,month,day,hour,minute,second,microsecond to XTIMESTAMP (UNIXTIME in microsecond precision)

Parameters

secmsec – list of year,month,day,hour,minute,second,microsecond

Returns

number of microseconds since the epoch 1970-01-01 00:00:00.000000

adapya.base.dtconv.xtimestamp2timestamp(mics)

>>> xtimestamp2timestamp(0)
(1970, 1, 1, 0, 0, 0, 0)

adapya.base.dtconv.xts2utc(microseconds)

same as micro2utc() only with epoch 1970-01-01

dump - Buffer hex display functions

This module contains buffer storage display functions

• most notably dump() used for logging buffers

• classes/methods for reversing readable SYSUDUMP data into virtual storage

• diffdump for printing differences in buffers using difflib

adapya.base.dump.diffbin(a, b, header1='Buffer1', header2='Buffer2', startaddr=0, fd=None)

Binary difference printer for buffer objects a and b

adapya.base.dump.diffdump(buf1, buf2, header1='Buffer 1', header2='Buffer 2', startaddr=None, fd=None)

Print the difference of 2 buffers

adapya.base.dump.dump(buf, header='Buffer', prefix='', startaddr=None, fd=None, log=None, ecodec='cp037')

print or log buffer

Parameters

• buf – string, bytestring or bytearray

• ecodec – ebcdic codec name for ebcdic printable characters

• startaddr – dump lines will be prefixed with given start address plus offset as absolute address rather than relative offset from buffer start

Example to write warning to logger adalog:

dump(a,log=adalog.warning)

adapya.base.dump.hex2buf(hexbuf)

convert hexadecimal input ‘hexbuf’ to buffer string blanks and CR LF are suppressed

adapya.base.dump.hex3buf(hexbuf)

convert TSO interpreted hex screen to buffer input must contain triple of interpreted and 2 hex lines where each column is one character

adapya.base.dump.hexRbuf(hexbuf)

interpret printed hex printed buffers in various formats formats: TSO print

adapya.base.dump.ishexstr(str)

test string str for being all hexadecimal characters

adapya.base.dump.txt2buf(txt)

convert text input ‘txt’ that may be split over several lines to string blanks and CR LF are suppressed

ecscodec - mapping ECS encodings to Python codecs

ECS is the acronym for Entire Conversion Services, a text conversion library used with Adabas.

Note

currently not all ECS encodings and Python codecs are listed. Non-existent codepage numbers could be mapped to the Python encodings per default ‘cp%d’ % i

adapya.base.ecscodec.getcodec(ecskey)

return the Python codec name given the ECS encoding key

adapya.base.ecscodec.getecs(codec)

return the ECS encoding key given the Python codec name

ftptoolz module

The Ftpzos class in ftptoolz contains z/OS specific methods for accessing the SPOOL and datasets.

Jobs

• submit()

• listjobs()

• listjob()

• deletejob()

• getjoblines()

PDS dataset

• memberinfo()

• memberlines()

• members()

• getbinaryfile()

• getfile()

• getlines()

• storemember()

• touchmembers()

ftptoolz modules requires Python V2.7 or higher

class adapya.base.ftptoolz.Dataset(volume, unit, referred, extents, used, recfm, lrecl, blksize, dsorg, dsname)

The namedtuple Dataset describes the information obtained from the z/OS ftp directory listing

>>> Dataset('VSM033','3390','20120215',1,10,'U',6447,6447,'PO','ABC.LOAD')

Example directories:

ARCIVE Not Direct Access Device                         VLOG.G0001V00
Dataset('ARCIVE','','',1,10,'',0,0,'','VLOG.G0001V00')
                                                   GDG  NEW.VLOG
VSM002 3390   2012/09/12  1   15  VB   27994 27998  PS  NEW.VLOG.G0115V00
VSM123 3390   **NONE**    1   45  ?        0     0 NONE NEW.VLOG.G0120V00
                                                   VSAM MOON.DDIR
VSM007 3390                                        VSAM MOON.DDIR.D

blksize

Alias for field number 7

dsname

Alias for field number 9

dsorg

Alias for field number 8

extents

Alias for field number 3

lrecl

Alias for field number 6

recfm

Alias for field number 5

referred

Alias for field number 2

unit

Alias for field number 1

used

Alias for field number 4

volume

Alias for field number 0

exception adapya.base.ftptoolz.DatasetsPDSspecifiedError

class adapya.base.ftptoolz.Ftps(host='', user='', passwd='', acct='', keyfile=None, certfile=None, context=None, timeout=<object object>, source_address=None)

Explicit FTPS, with shared TLS session (needs Python > 3.6 ?) to avoid ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2746) see https://stackoverflow.com/questions/14659154/ftpes-session-reuse-required

ntransfercmd(cmd, rest=None)

Initiate a transfer over the data connection.

If the transfer is active, send a port command and the transfer command, and accept the connection. If the server is passive, send a pasv command, connect to it, and start the transfer command. Either way, return the socket for the connection and the expected size of the transfer. The expected size may be None if it could not be determined.

Optional `rest’ argument can be a string that is sent as the argument to a REST command. This is essentially a server marker used to tell the server to skip over any data up to the given marker.

class adapya.base.ftptoolz.Ftpzos(host, user, passwd, verbose=0, test=0, timeout=None, certfile='', encoding='latin1')

wraps ftplib.FTP with specific z/OS functions like submitting jobs and reading JES spool

>>> from tools.ftptoolz import Ftpzos
>>> ftp = Ftpzos('host','user','password',verbose=2)

datasets(dsnprefix)

generator function yields catalog information for each dataset found under prefix returning a Dataset() tuple

Parameters

dsnprefix – read file or pds member

deletejob(jobid)

Delete Job identified by jobid

Parameters

jobid – e.g. ‘JOB12345’

getbinaryfile(filename, outfile, rdw=0)

Copy binary file from ftp to file

Parameters

• filename – read file or pds member

• outfile – output file to receive contents

• rdw – if true transfers file with variable records being prefixed with the RDW word (Int2 record length inclusive, 2 bytes dummy)

getfile(filename, outfile, encoding=None, xlate='', cmp=False)

copy text file from ftp to file

Parameters

• filename – read file or pds member

• outfile – optional output file to receive contents

• encoding – encoding for writing outfile

• xlate – convert data from EBCDIC to ASCII (option) specify name for dataset of the form hlq.<xlate>.TCPXLBIN

• cmp – if set to true: compare file gotten with the outfile

Returns

True if contents of file ‘filename’ same as file ‘outfile’ and cmp == True else False

getjob(jobid, outfile=None)

Copies job output to outfile

Parameters

• jobid – JOBID.X for complete spool output or JOBID.n for individual spool files e.g. JOB12345.X or JOB12345.1

• outfile – optional output file to receive contents

Returns filename

file name of output file

getjoblines(filename, outfile=None)

Generator function if no outfile specified

yields a line

Parameters

• filename – JOBID.X for complete spool output or JOBID.n for individual spool files e.g. JOB12345.X or JOB12345.1

• outfile – optional output file to receive contents

getlines(filename)

generator function yields a line read from sequential file or PDS member specified by filename

Parameters

filename – read file or pds member

listjob(jobid, jobowner='*', jobstat='ALL')

List specific job given by jobid :param jobid: job id e.g. ‘JOB12345’ :param jobowner: job owner :param jobstat: job status: one of ALL, OUTPUT or ACTIVE :returns: jobstatus namedtuple and spoolfiles namedtuples

memberinfo(dsn)

returns plain pds member directory listing for .mirrorinfo

memberlines(dsn)

Generator function returns the member lines of a PDS (= directory list) :param dsn: dataset name of a PDS or PDSE

members(dsn)

Generator function returns Member namedtuple (name, changed, size, userid) :param dir: dataset name of a PDS or PDSE

pdsnames(dsnprefix, recfm='FB')

Iterate over datasets starting with prefix and return only PDS or PDSE datasets

Parameters

• dsnprefix – Dataset prefix - single quoted

• recfm – filter special datasets (e.g. ‘?B’ selects ‘FB’ or VB’

storemember(pds, fname)

upload file to pds member :param pds: partitioned dataset name :param fname: file in current directory to be uploaded to pds

submit(f)

submit job

Parameters

f – opened file object with JCL to submit Must be bytes type data (Latin1) in Python 3

Return jobid

job number string e.g. JOB12345 or ‘’ of submit failed

submitWait(jcl, wait=30)

Parameters

• jcl – dataset or pds(member) containting JCL to submit

• wait – wait time in seconds until function is to return

Return Job

Job object containing information on Job submitted or None

>>> fz=ftptoolz.Ftpzos('zibm','mm','pw',verbose=2)
>>> j=fz.submitWait("'mm.jobs(copyy)'")
>>> x.cc
'AE37'

touchmembers(pds, membertimes, touchtemplate)

Submit TOUCH job to set modification times in members of a

partitioned dataset.

Parameters

• pds – partitioned dataset name

• membertimes – list of (membername, modtime, uid, size) tuples modtime is of datetime type or of string ‘yyyymmdd.HHMMSS’

• touchtemplate – Touch template job skeleton (see touchtemplate_sample for further details

class adapya.base.ftptoolz.Job(jobid, jcl)

stores the JES job information

class adapya.base.ftptoolz.Jobstatus(jobname, jobid, owner, status, cl, cc, numspool, cputime, elapsed, stepname, procname)

The namedtuple Jobstatus describes the jobstatus in z/OS as returned from ftp

>>> Jobstatus('MMP8','JOB23114','ACF2STC','OUTPUT','K','0000',5)
>>>                                                 or 'A000' if abend

cc

Alias for field number 5

cl

Alias for field number 4

cputime

Alias for field number 7

elapsed

Alias for field number 8

jobid

Alias for field number 1

jobname

Alias for field number 0

numspool

Alias for field number 6

owner

Alias for field number 2

procname

Alias for field number 10

status

Alias for field number 3

stepname

Alias for field number 9

class adapya.base.ftptoolz.Member(name, changed, size, userid)

The namedtuple Member describes the member information in a directory listing of a partitioned dataset (PDS).

changed

Alias for field number 1

name

Alias for field number 0

size

Alias for field number 2

userid

Alias for field number 3

class adapya.base.ftptoolz.ReusedSslSocket(*args, **kwargs)

Additional fix https://bugs.python.org/issue31727

unwrap()

Start the SSL shutdown handshake.

class adapya.base.ftptoolz.Spoolfile(id, stepname, procstep, cl, ddname, numbytes)

The namedtuple Spoolfile describes the a JES spool file in z/OS as returned from ftp

>>> Spoolfile(1,'JES2','STEP1','X','JESJCL',7362)

cl

Alias for field number 3

ddname

Alias for field number 4

id

Alias for field number 0

numbytes

Alias for field number 5

procstep

Alias for field number 2

stepname

Alias for field number 1

adapya.base.ftptoolz.getasmdate(line)

find execution date in a line of an assembly or binder listing :param line: :returns date: date is a string of format YYYYMMDD.hhmm00

adapya.base.ftptoolz.writedict(dict, filename)

Write a dictionary to a file in a way that can be read back using rval() but is still somewhat readable (i.e. not a single long line). Also creates a backup file.

adapya.base.ftptoolz.writejson(dict, filename)

Write a dict as json file

jconfig - Manage configuration data in a JSON file

The default file name is .ztools located in the USERPROFILE/APPDATA/HOME directory.

adapya.base.jconfig.CFGFN = '.ztools'

default config file name

adapya.base.jconfig.SHOWCONFIG = 1

indicator to request display of the settings in getparms() and setparms()

adapya.base.jconfig.get(name, cf='.ztools')

get subject from config JSON file :param name: subject dictionary :param cf: config file name

adapya.base.jconfig.getparms(subject, show, cf='.ztools', **parms)

get for a subject configuration data unless provided by parameter values in parms :param subject: select parameters by subject e.g. ‘ftp’ :param parms: parameters that are requested :param show: if > 0: print parameters

>>> setparms('ftp',False,cf='.test',host='big',password='secret',user='hugo')
>>> getparms('ftp',False,cf='.test',host='',password='',user='')
{'host': 'big', 'password': 'secret', 'user': 'hugo'}

adapya.base.jconfig.set(name, value, cf='.ztools')

set subject in config JSON file :param name: subject dictionary :param cf: config file name

adapya.base.jconfig.setparms(subject, show, cf='.ztools', **parms)

Set configuration parameters for a subject

Parameters

• subject – select parameters by subject e.g. ‘ftp’

• parms – parameter that should be set If the value of a parameter is None it is ignored. This simplifies setting up call: no dynamic creation of parms dictionary needed.

• show – if True print parameters

>>> setparms('ftp',True,cf='.test',user='Anna',password='H2o',host='mojave')
The following configuration is stored in '.test' for 'ftp':
           host: mojave
       password: *password*
           user: Anna

recordio - read and write records

The module recordio contains functions to read or write records with special structures (variable, variable blocked):

• RDW records preceded by a record descriptor word

• BDW variable records blocked

• EXCL4 records preceded by a 4 bytes exclusive record length

  in native byte-order)

adapya.base.recordio.readrec(f, recform='', dumphdr='', numrec=0, skiprec=0, ecodec='cp037', debug=0, into=0)

readrec - Generator function to read records with special record format specified in recform

param f

filehandle of open file

param recform

record format to process

• ‘RDW’ variable record format (2 bytes length, Network byte order)

  return record without RDW header (exclusive)

• ‘RDW+’ variable record format (2 bytes length, Network byte order)

  return record including RDW header

  Note: for segmented records rlen in RDW header is length

  of first segment and not the whole record

• ‘BDW’ variable record blocked: input includes Block Descriptor Word

  which is skipped return record without RDW header (exclusive)

• ‘BDW+’ same as BDW but return record including RDW header

• ‘EXCL4’ exclusive 4 bytes length, native byte order

param dumphdr

header text of record; if not empty: prints record

param ecodec

Ebcdic codec for character interpretation when dumping records

param debug

1 - print RDW information

param into

1 - read into and return modifyable bytearray rather that bytes (only for non segemented RDW files)

Example usage:

>> for rec in readrec(f,recform='RDW',dumphdr='my_records'):
>>    process(rec)

adapya.base.recordio.writerec(f, record, isn=None, recform='')

writerec - function to write records with special record format

Parameters

• f – filehandle of open file

• record – record string/bytearray to be written

• isn – prefix record with an 4 byte integer in Network byte order

• recform –

  record format to process:

  - 'RDW' variable record format:
  

  2 bytes length, 2 bytes emtpy in Network byte order

class adapya.base.smfrecordz.Smf(**kw)

class adapya.base.smfrecordz.Smf30(**kw)

class adapya.base.smfrecordz.Smf30cas(**kw)

class adapya.base.smfrecordz.Smf30id(**kw)

class adapya.base.smfrecordz.Smf30prf(**kw)

class adapya.base.smfrecordz.Smf30pss(**kw)

adapya.base.smfrecordz.dtime100(i)

return readable time since midnight 1/100 sec precision

adapya.base.smfrecordz.i2dt(idat, itim)

Convert industry date time to datetime object

Parameters

• idat – integer IBM industry date 0cyyddd

• itim – integer IBM time in 100ths of a second in day

adapya.base.smfrecordz.idate(i)

Return string from IBM date 0cyyddd

stck - STCK timestamp format conversion routines

The module stck.py contains functions for converting timestamp values in STCK format (used on IBM mainframe computers).

class adapya.base.stck.Utc

dst(dt)

datetime -> DST offset as timedelta positive east of UTC.

tzname(dt)

datetime -> string name of time zone.

utcoffset(dt)

datetime -> timedelta showing offset from UTC, negative values indicating West of UTC

adapya.base.stck.csec(stcksec)

returns seconds converted from stck seconds

>>> csec(0xd69)
3599.761408

adapya.base.stck.cstck(stck)

returns seconds_since_1970

adapya.base.stck.cstckd(stckd)

converts long STCK time into local time and microseconds

adapya.base.stck.leap4dt(dt)

Determine the leap seconds for a given datetime.

Parameters

dt – datetime value

Returns

leap seconds

>>> leap4dt(datetime(1972, 6,30, 23,59,59))
0
>>> leap4dt(datetime(2017, 1,1))
27

adapya.base.stck.sstck(stck, gmt=0)

returns ISO date time string from local stck if gmt !=0: GMT STCK is assumed

adapya.base.stck.sstckd(stckd, gmt=0)

converts long STCK time into string of local time and microseconds if gmt !=0: GMT STCK is assumed

adapya.base.stck.sstckgmt(stck)

returns ISO date time string assuming gmt stck value

adapya.base.stck.stimet(timet)

Convert time_t value into datetime string, allows negative values.

Supports datetimes between 1900-01-01 and 9999-12-31

>>> stimet(-(70*365+17)*24*3600)
'1900-01-01 00:00:00 UTC (+0000)'

>>> stimet((8035*365+121)*24*3600+23*3600+3599)
'9999-12-31 23:59:59 UTC (+0000)'

adapya.base.stck.utc2stckd(dt=datetime.datetime(2023, 12, 7, 21, 25, 53, 567696), leapsec=False)

convert a datetime to STCK format

Parameters

• dt – datetime value to convert to STCK (detault now)

• leapsec – if True add in leap seconds relevant for the datetime dt

touch - set modification time of file(s)

The program works in 2 ways:

• with option -m:

  Find .mirrorinfo in current directory or subdirectory for each file in sub-/directory get its modifcation date from .mirrorinfo and set the file to this date

  • This can be used after SVN checkout where the mod. dates need to be reset from the checkin to the original date

  • The .mirrorinfo is created by the ftpmirroz.py ftpz.py programs a default extension ‘.s’ is expected for the local files for comparison with the members listed in .mirrorinfo the extension is removed and the filename is upper cased

• with file_name and optional date/time:

  Sets file to current or given date/time

Usage:

touch.py file_name [yyyy-mm-dd [HH:MM:SS]]

                Other date/time forms:
                   date:  yyyy/mm/dd
                   time:  HH:MM

touch.py [-m] [-v] [-x .ext]

-m mirrorinfo  set dates according to .mirrorinfo
               Either .mirrorinfo is on current directory or
               in next lower subdirectory
-v verbose
-x extension   default extension is '.s'

adapya.base.touch.ftouch(file, newtime)

Set new modified time for given file. Supports Unix and Windows (with win32 installed)

Parameters

• file – file to be modified

• newtime – new time stamp to be set

adapya.base.touch.get_mirrorinfo_dict(path)

Read mirrorinfo. file containing the PDS directory and return it as dictionary. The .mirrorinfo file is produced by ftpmirroz.py)

Parameters

path – path without file name

Return info

dictionary with PDS directory lines for each member key. If not .mirrorinfo not found return empty dict