An Application

This is a declarative structure written in either JSON or YAML. YAML is more compact and makes it easier to add multi-line documentation. With the command-line interface you can convert from JSON to YAML and back.

The top-level of an application has at least the application and parts fields. The application name should be unique within the cluster. The parts form the structure of the application. Optionally the fields version, parameters, title and description can be added. All are strings.

This is a simple application with just one stream. It consumes the Kafka topic my-commands and retains only those messages that have the _command field set to put. The remaining messages are written to the Kafka topic my-commands-filtered.

---
application: "my-app"
version: "1.0"
parts:
  - type: "stream"
    name: "my-stream"
    fromTopic: "my-commands"
    toTopic: "my-commands-filtered"
    pipeline:
      - $match:
          _command: "put"

We could have left out the toTopic field. In that case the results are available as the stream called my-stream. Another part of the application can consume that stream by referring to its name. It could look like this:

---
application: "my-app"
version: "1.0"
parts:
  - type: "stream"
    name: "my-stream"
    fromTopic: "my-commands"
    pipeline:
      - $match:
          _command: "put"
  - type: "stream"
    name: "set-filtered"
    fromStream: "my-stream"
    toTopic: "my-commands-filtered"
    pipeline:
      - $addFields:
          filtered: true    

Now the second part takes the output of the first one and adds the filtered field to each message before writing it to the topic my-commands-filtered. Of course, in practice you would add the $addFields pipeline stage to the first part, right after the $match stage.

Sometimes a file can grow very large when there are many parts. Then it is more convenient to move the parts in separate files like this:

---
application: "my-app"
version: "1.0"
parts:
  - "my_stream.yml"
  - "set_filtered.yml"

The files would then contain the following respectively:

---
type: "stream"
name: "my-stream"
fromTopic: "my-commands"
pipeline:
  - $match:
      _command: "put"

---
type: "stream"
name: "set-filtered"
fromStream: "my-stream"
toTopic: "my-commands-filtered"
pipeline:
  - $addFields:
      filtered: true    

Part files can also contain arrays of parts. Everything will be loaded and assembled into one array of parts. Several applications can be combined in one array as well. Each entry will still be treated as a separate application. This is an example:

---
- application: "my-app1"
  version: "1.0"
  parts:
    - "parts1.yml"
- application: "my-app2"
  version: "1.0"
  parts:
    - "parts2.yml"