We propose use of sparsemax based transformers which compared to vanilla transformers generate sparsity in attention mechanism.