Tsumugi

A modern PySpark Connect/Classic wrapper on top of the Deequ, a beautiful Data Quality library from AWS Labs.

NOTE: Tsumugi Shiraui is a chimera: a hybrid of Human and Gauna. She combines the chaotic power of Gauna with a Human intillegence and empathia. Like an original character of the Manga "Knights of Sidonia", this project aims to make a hybrid of very powerful but hard to learn and use Deequ Scala Library with a usability and simplicity of Spark Connect (PySpark Connect, Spark Connect Go, Spark Connect Rust, etc.).

Table on contents

• Lungauge agnostic concepts and usecases for Deequ
  • Concepts of Deequ
  • Possible usecases for Deequ
• PySpark Connect / Classic API
  • Main data structures and classes
  • API Docs (auto-generated)
• Example notebooks
  • Basic example
  • Using predicates with analyzers