Map Operator

The Map Operator is a versatile tool that allows you to transform an existing message and re-encode the input into a new message format. The operator takes the input and maps it into any type. Every message in the original topic will be mapped to a new topic. Please see filter-mapping for filter and mapping.

Prerequisites

This guide uses local Fluvio cluster. If you need to install it, please follow the instructions at here.

Syntax

Below is an example of a tranform function. The example demostrates how to map a sentence into a new string that outputs the size of the original string.

    transforms:
      - operator: map 
        run: |
          fn get_sentence_length(input: String) -> Result<String> {
            Ok(format!("Inserted string size {}",input.len()))
          }

Running the Example

Copy and paste following config and save it as dataflow.yaml.

# dataflow.yaml
apiVersion: 0.5.0
meta:
  name: map-example
  version: 0.1.0
  namespace: examples

config:
  converter: raw

topics:
  sentences:
    schema:
      value:
        type: string
  len:
    schema:
      value:
        type: string

services:
  map-service:
    sources:
      - type: topic
        id: sentences
    transforms:
      - operator: map 
        run: |
          fn get_sentence_length(input: String) -> Result<String> {
            Ok(format!("Inserted string size {}",input.len()))
          }
    sinks:
      - type: topic
        id: len

To run example:

$ sdf run --ephemeral

Produce sentences to in sentence topic:

$ echo "This string is size 22" | fluvio produce sentences
$ echo "This string is size 23." | fluvio produce sentences

Consume topic len to retrieve the result in another terminal:

$ fluvio consume len -Bd
Inserted string size 22
Inserted string size 23

Each string will be mapped via the function provided.

Cleanup

Exit sdf terminal and clean-up. The --force flag removes the topics:

$ sdf clean --force

Conclusion

We just covered another basic operator in SDF, the Map Operator.