gt4sd.frameworks.granular.ml.models.module module

Generic modules.

Summary

Classes:

ConvBottleneck	Set of convolutional layers to reduce memory matrix to single latent vector.
DeconvBottleneck	Set of deconvolutional layers to reshape latent vector back into memory matrix.
Embeddings	Transforms input token id tensors to size d_model embeddings.
LayerNorm	Custom layer normalization.
ListModule	Create single pytorch module from list of modules.
Mlp	MLP module.
MlpDecoder	MLP decoder.
MlpEncoder	MLP encoder.
MultiHeadedAttention	Multihead attention implementation (based on Vaswani et al.).
PositionalEncoding	Static sinusoidal positional encoding layer.
PositionwiseFeedForward	Feed forward implementation.
RnnDecoder	RNN decoder.
RnnEncoder	RNN encoder.
SublayerConnection	A residual connection followed by a layer normalization.
TorchLayerNorm	Layer normalization using torch BatchNorm1d.
TransformerDecoder	Base transformer decoder architecture.
TransformerDecoderLayer	Self-attention/source-attention/feedforward implementation.
TransformerEncoder	Base transformer encoder architecture.
TransformerEncoderLayer	Self-attention/feedforward implementation.

Functions:

attention	Compute scaled dot product attention (adapted from Viswani et al.).
clones	Produce N identical layers (adapted from http://nlp.seas.harvard.edu/2018/04/03/attention.html).
subsequent_mask	Mask out subsequent positions (adapted from http://nlp.seas.harvard.edu/2018/04/03/attention.html).

Reference

class Mlp(input_size, hidden_size, output_size, n_layers, activation, dropout, **kwargs)

Bases: Module

MLP module.

__init__(input_size, hidden_size, output_size, n_layers, activation, dropout, **kwargs)

Construct Mlp.

Parameters

• input_size (int) – size of the input.

• hidden_size (int) – size of the hidden layers.

• output_size (int) – size of the output.

• n_layers (int) – number of layers.

• activation (str) – name of the activation.

• dropout (float) – dropout rate.

forward(x)

Forward pass in the model.

Parameters

x (Tensor) – model input.

Return type

Tensor

Returns

model output.

__annotations__ = {}

__doc__ = 'MLP module.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class MlpEncoder(input_size, hidden_size, output_size, n_layers, activation, dropout=0.0, **kwargs)

Bases: Mlp

MLP encoder.

__init__(input_size, hidden_size, output_size, n_layers, activation, dropout=0.0, **kwargs)

Construct MlpEncoder.

Parameters

• input_size (int) – size of the input.

• hidden_size (int) – size of the hidden layers.

• output_size (int) – size of the output.

• n_layers (int) – number of layers.

• activation (str) – name of the activation.

• dropout (float) – dropout rate. Defaults to 0.0.

__annotations__ = {}

__doc__ = 'MLP encoder.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class MlpDecoder(latent_size, hidden_size, output_size, n_layers, activation, dropout=0.0, **kwargs)

Bases: Mlp

MLP decoder.

__init__(latent_size, hidden_size, output_size, n_layers, activation, dropout=0.0, **kwargs)

Construct MlpEncoder.

Parameters

• latent_size (int) – size of the input.

• hidden_size (int) – size of the hidden layers.

• output_size (int) – size of the output.

• n_layers (int) – number of layers.

• activation (str) – name of the activation.

• dropout (float) – dropout rate. Defaults to 0.0.

__annotations__ = {}

__doc__ = 'MLP decoder.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class RnnEncoder(vocab_size, embedding_size, hidden_size=256, n_layers=2, bidirectional=False, latent_size=196)

Bases: Module

RNN encoder.

__init__(vocab_size, embedding_size, hidden_size=256, n_layers=2, bidirectional=False, latent_size=196)

Construct RnnEncoder.

Parameters

• vocab_size (int) – size of the vocabulary.

• embedding_size (int) – size of the embedding vectors.

• hidden_size (int) – hidden size. Defaults to 256.

• n_layers (int) – number of layers. Defaults to 2.

• bidirectional (bool) – whether the RNN cell is bidirectional. Defaults to False.

• latent_size (int) – latent size. Defaults to 196.

forward(input_sequence)

Forward pass in the model.

Parameters

input_sequence (Tensor) – input sequence tensor.

Return type

Tuple[Tensor, Tensor]

Returns

a tuple containing hidden state and embedded sequence.

__annotations__ = {}

__doc__ = 'RNN encoder.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class RnnDecoder(vocab_size, embedding_size, hidden_size=256, n_layers=2, latent_size=196)

Bases: Module

RNN decoder.

__init__(vocab_size, embedding_size, hidden_size=256, n_layers=2, latent_size=196)

Construct RnnDecoder.

Parameters

• vocab_size (int) – size of the vocabulary.

• embedding_size (int) – size of the embedding vectors.

• hidden_size (int) – hidden size. Defaults to 256.

• n_layers (int) – number of layers. Defaults to 2.

• latent_size (int) – latent size. Defaults to 196.

forward(latent, input_embedding)

Forward pass in the model.

Parameters

• latent (Tensor) – latent tensor.

• input_embedding (Tensor) – input embedding.

Return type

Tensor

Returns

model output.

inference_direct(latent, embedding, tokenizer, max_len)

Direct inference from latent space.

Parameters

• latent (Tensor) – latent tensor.

• embedding (Module) – embedding module.

• tokenizer (Tokenizer) – tokenizer.

• max_len (int) – maximum sequence length.

Return type

Tuple[List[str], Tensor]

Returns

a tuple containing decoded strings and indices.

__annotations__ = {}

__doc__ = 'RNN decoder.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

attention(query, key, value, mask=None, dropout=None)

Compute scaled dot product attention (adapted from Viswani et al.).

Parameters

• query (Tensor) – query tensor.

• key (Tensor) – key tensor.

• value (Tensor) – value tesor.

• mask (Optional[Tensor, None]) – mask to apply on attention score. Defaults to None, a.k.a., no mask.

• dropout (Optional[Module, None]) – dropout layer. Defaults to None, a.k.a., no dropout.

Return type

Tuple[Tensor, Tensor]

Returns

a tuple containing the applied attention and the attention weights.

clones(module, n)

Produce N identical layers (adapted from http://nlp.seas.harvard.edu/2018/04/03/attention.html).

Parameters

• module (Module) – a module.

• n (int) – number of clones.

Return type

Module

Returns

a module list.

subsequent_mask(size)

Mask out subsequent positions (adapted from http://nlp.seas.harvard.edu/2018/04/03/attention.html).

Parameters

size (int) – size of the attention matrix.

Return type

Tensor

Returns

the mask tensor.

class ListModule(*args)

Bases: Module

Create single pytorch module from list of modules.

__init__(*args)

Construct ListModule.

__getitem__(idx)

Get item from the module list.

Parameters

idx (int) – index of the item.

Raises

IndexError – in case the index is out of range.

Return type

Any

Returns

the item.

__iter__()

An iterator over the module list values.

Return type

Any

Returns

the iterator over values.

__len__()

Length of the module list.

Returns

the number of modules.

__annotations__ = {}

__doc__ = 'Create single pytorch module from list of modules.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class MultiHeadedAttention(h, d_model, dropout=0.1)

Bases: Module

Multihead attention implementation (based on Vaswani et al.).

__init__(h, d_model, dropout=0.1)

Construct MultiHeadedAttention.

Parameters

• h – number of heads.

• d_model – model size.

• dropout – dropout rate. Defaults to 0.1.

forward(query, key, value, mask=None, return_attn=False)

Forward pass in the model.

Parameters

• query (Tensor) – query tensor.

• key (Tensor) – key tensor.

• value (Tensor) – value tesor.

• mask (Optional[Tensor, None]) – mask to apply on attention score. Defaults to None, a.k.a., no mask.

• return_attn (bool) – whether to return the attention matrix instead of the linear layer output. Defaults to False, a.k.a, do not return attention.

Return type

Any

Returns

either the last layer output of the attention matrix.

__annotations__ = {}

__doc__ = 'Multihead attention implementation (based on Vaswani et al.).'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class PositionwiseFeedForward(d_model, d_ff, dropout=0.1)

Bases: Module

Feed forward implementation.

__init__(d_model, d_ff, dropout=0.1)

Construct PositionwiseFeedForward.

Parameters

• d_model (int) – model size.

• d_ff (int) – feed forward size.

• dropout (float) – dropout rate. Defaults to 0.1.

forward(x)

Forward pass in the model.

Parameters

x (Tensor) – input tensor.

Return type

Tensor

Returns

feed forward output.

__annotations__ = {}

__doc__ = 'Feed forward implementation.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class ConvBottleneck(size, number_of_layers=3)

Bases: Module

Set of convolutional layers to reduce memory matrix to single latent vector.

__init__(size, number_of_layers=3)

Construct ConvBottleneck.

Parameters

• size (int) – input size.

• number_of_layers (int) – convolutional layers number. Defaults to 3.

forward(x)

Forward pass in the model.

Parameters

x (Tensor) – input tensor.

Return type

Tensor

Returns

model output.

__annotations__ = {}

__doc__ = 'Set of convolutional layers to reduce memory matrix to single latent vector.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class DeconvBottleneck(size, seq_len, dim_factor)

Bases: Module

Set of deconvolutional layers to reshape latent vector back into memory matrix.

__init__(size, seq_len, dim_factor)

Construct DeconvBottleneck.

Parameters

• size (int) – size of the deconvolutional padding.

• seq_len (int) – length of the sequence.

• dim_factor (int) – dimensionality factor.

forward(x)

Forward pass in the model.

Parameters

x (Tensor) – input tensor.

Return type

Tensor

Returns

model output.

__annotations__ = {}

__doc__ = 'Set of deconvolutional layers to reshape latent vector back into memory matrix.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class Embeddings(d_model, vocab_size)

Bases: Module

Transforms input token id tensors to size d_model embeddings.

__init__(d_model, vocab_size)

Costruct Embeddings.

Parameters

• d_model (int) – size of the embedding vectors.

• vocab_size (int) – size of the vocabulary.

forward(x)

Forward pass in the model.

Parameters

x (Tensor) – input tensor.

Return type

Tensor

Returns

model output.

__annotations__ = {}

__doc__ = 'Transforms input token id tensors to size d_model embeddings.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class PositionalEncoding(d_model, dropout, max_len=5000)

Bases: Module

Static sinusoidal positional encoding layer.

__init__(d_model, dropout, max_len=5000)

Construct PositionalEncoding.

Parameters

• d_model (int) – model size.

• dropout (float) – dropout rate.

• max_len (int) – maximum sequence length. Defaults to 5000.

forward(x)

Forward pass in the model.

Parameters

x (Tensor) – input tensor.

Return type

Tensor

Returns

model output.

__annotations__ = {}

__doc__ = 'Static sinusoidal positional encoding layer.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class TorchLayerNorm(features, eps=1e-06)

Bases: Module

Layer normalization using torch BatchNorm1d.

__init__(features, eps=1e-06)

Construct TorchLayerNorm.

Parameters

• features (int) – number of features.

• eps – espilon to add to denominator for numerical stability. Defaults to 1e-6.

forward(x)

Forward pass in the model.

Parameters

x (Tensor) – input tensor.

Return type

Tensor

Returns

model output.

__annotations__ = {}

__doc__ = 'Layer normalization using torch BatchNorm1d.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class LayerNorm(features, eps=1e-06)

Bases: Module

Custom layer normalization.

__init__(features, eps=1e-06)

Construct LayerNorm.

Parameters

• features (int) – number of features.

• eps – espilon to add to denominator for numerical stability. Defaults to 1e-6.

forward(x)

Forward pass in the model.

Parameters

x (Tensor) – input tensor.

Return type

Tensor

Returns

model output.

__annotations__ = {}

__doc__ = 'Custom layer normalization.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class SublayerConnection(size, dropout)

Bases: Module

A residual connection followed by a layer normalization.

Note for code simplicity the norm is first as opposed to last. A dropout layer is also applied.

__init__(size, dropout)

Initializes internal Module state, shared by both nn.Module and ScriptModule.

forward(x, sublayer)

Forward pass in the model.

Parameters

• x (Tensor) – input tensor.

• sublayer (Callable) – a callable returning a tensor.

Return type

Tensor

Returns

model output.

__annotations__ = {}

__doc__ = 'A residual connection followed by a layer normalization.\n\n    Note for code simplicity the norm is first as opposed to last. A dropout layer\n    is also applied.\n    '

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class TransformerEncoder(hidden_size, ff_size, seq_len, dropout, heads, n_layers_enc, vocab_size, bypass_bottleneck)

Bases: Module

Base transformer encoder architecture.

__init__(hidden_size, ff_size, seq_len, dropout, heads, n_layers_enc, vocab_size, bypass_bottleneck)

Construct TransformerEncoder.

Parameters

• hidden_size (int) – hidden size.

• ff_size (int) – feed forward size.

• seq_len (int) – sequence length.

• dropout (float) – dropout rate.

• heads (int) – number of heads.

• n_layers_enc (int) – number of encoding layers.

• vocab_size (int) – vocabulary size.

• bypass_bottleneck (bool) – whether the bottleneck should be by passed.

calc_output_shape(seq_len, hidden_size)

Compute output shape.

Parameters

• seq_len (int) – sequence length.

• hidden_size (int) – hidden size.

Returns

convolutional bottleneck output shape.

forward(x, mask)

Forward pass in the model.

Parameters

• x (Tensor) – input tensor.

• mask (Tensor) – mask to apply in the attention layer.

Return type

Tensor

Returns

model output.

__annotations__ = {}

__doc__ = 'Base transformer encoder architecture.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class TransformerEncoderLayer(size, seq_len, self_attn, feed_forward, dropout)

Bases: Module

Self-attention/feedforward implementation.

__init__(size, seq_len, self_attn, feed_forward, dropout)

Construct TransformerEncoderLayer.

Parameters

• size (int) – model size.

• seq_len (int) – sequence length.

• self_attn (Module) – self-attention layer.

• feed_forward (Module) – feed forward layer.

• dropout (float) – droupout rate.

forward(x, mask, return_attn=False)

Forward pass in the model.

Parameters

• x (Tensor) – input tensor.

• mask (Tensor) – mask to apply in the attention layer.

• return_attn (bool) – whether to return the attention together with the output. Defaults to False, return only encoder output.

Return type

Any

Returns

model output.

__annotations__ = {}

__doc__ = 'Self-attention/feedforward implementation.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class TransformerDecoder(hidden_size, ff_size, seq_len, dropout, heads, n_layers_dec, latent_size, vocab_size, bypass_bottleneck, deconv_shape)

Bases: Module

Base transformer decoder architecture.

__init__(hidden_size, ff_size, seq_len, dropout, heads, n_layers_dec, latent_size, vocab_size, bypass_bottleneck, deconv_shape)

Construct TransformerDecoder.

Parameters

• hidden_size (int) – hidden size.

• ff_size (int) – feed forward size.

• seq_len (int) – sequence length.

• dropout (float) – dropout rate.

• heads (int) – number of heads.

• n_layers_enc – number of encoding layers.

• latent_size (int) – latent size.

• vocab_size (int) – vocabulary size.

• bypass_bottleneck (bool) – whether the bottleneck should be by passed.

• deconv_shape (Tuple[int, int, int]) – shape of the deconvoluted samples. A tuple with three dimensions.

forward(x, mem, src_mask, tgt_mask)

Forward pass in the model.

Parameters

• x (Tensor) – input tensor.

• mem (Tensor) – memory tensor.

• src_mask (Tensor) – source sequence mask.

• tgt_mask (Tensor) – target sequence mask.

Return type

Tensor

Returns

model output.

inference_direct(latent, mask_lengths, tokenizer)

Direct inference from latent space.

Parameters

• latent (Tensor) – latent tensor.

• mask_lengths (Tensor) – masking tensor.

• tokenizer (Tokenizer) – tokenizer.

Return type

Tuple[List[str], Tensor]

Returns

a tuple containing decoded strings and indices.

__annotations__ = {}

__doc__ = 'Base transformer decoder architecture.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'

class TransformerDecoderLayer(size, seq_len, self_attn, src_attn, feed_forward, dropout)

Bases: Module

Self-attention/source-attention/feedforward implementation.

__init__(size, seq_len, self_attn, src_attn, feed_forward, dropout)

Construct TransformerDecoderLayer.

Parameters

• size (int) – model size.

• seq_len (int) – sequence length.

• self_attn (Module) – self-attention layer.

• src_attn (Module) – source attention layer.

• feed_forward (Module) – feed forward layer.

• dropout (float) – droupout rate.

forward(x, memory_key, memory_val, src_mask, tgt_mask, return_attn=False)

Forward pass in the model.

Parameters

• x (Tensor) – input tensor

• memory_key (Tensor) – memory key tensor.

• memory_val (Tensor) – memory value tensor.s

• src_mask (Tensor) – mask to apply in the source attention layer.

• tgt_mask (Tensor) – mask to apply in the target attention layer.

• return_attn (bool) – whether to return the attention together with the output. Defaults to False, return only encoder output.

Return type

Any

Returns

model output.

__annotations__ = {}

__doc__ = 'Self-attention/source-attention/feedforward implementation.'

__module__ = 'gt4sd.frameworks.granular.ml.models.module'