Kim Pham

rustling in the open source maplibre community

Posted on Feb 11, 2023

starting in september i spent a few months learning rust. i had a few inchoate reasons:

  • i’ve always wanted to learn systems programming…i wanted to be exposed to deeper programming concepts. i do appreciate web development and working on web and business logic on the front and backend, but i’m also interested in coding that’s closer to computer science, or at least working with the ideas of computer science. and to better understand things at a lower level. rip off the hood of abstraction
  • people talk about it a lot on hacker news and in the blockchain space, i know you know. but also
  • linux is supporting rust. cool! so i guess it’s it’s here to stay

so then i learned it https://github.com/kimpham54/rust-book-homework-kim. i was pretty motivated because i was trying to get into the january polkadot blockchain academy in buenos aires, and got through most of the rust book. unfotunately i didn’t get a chance to take the rust exam… so now what? well i really enjoyed learning it, so might as well put those amateur skills to use.

to me, contributing to open-source is a beautiful and brutal mechanism for self-reflection for both personal professional growth. the transparency, the feedback, the meritocratic initiative, it’s a chance to discover your intrinsic/extrinsic motivations and to know who you are, who you should be, and who you want to be as a worker. i don’t know of a better way to know what you truly know/don’t know about coding. you have to apply your skills in front of a surveying audience and do it without any obvious incentive. teaching comes close to this naked vulnerability feeling, but even then it masks a bit of the truth. it’s humbling, empowering. ok i’m waxing poetic a bit here, but i’m sentimental because in the past my most rewarding and idealized professional experiences were when I got to work in open source. I do miss being part of that more in libraryland world - much love forever goes to Islandora.

so i looked for some open source rust projects, and found one that hits close to my passion for maps - maplibre-rs. i reached out to max, who’s heading development on maplibre-rs. i’ve lurked in the community for a bit and let him know i was interested in a beginner task. so finally when one came up i volunteered.

i was working on parsing some style specs over the christmas break. took a break, then came back to it in february, which means i forgot everything. but it’s also cool because i look at my code and go whoa, how did i know how to do this?

henceforth i shall keep notes, and you’re welcome to peruse them too (cw: may not make sense)!

maplibre-rs style parser

i’m only concerned with the style module in maplibre-rs. here’s how the project is organized

maplibre-rs/maplibre/src/style
├── layer.rs
├── mod.rs
├── original
│   ├── layer.rs
│   ├── source.rs
│   └── style.rs
├── source.rs
├── style.rs
└── tests
    ├── map-styles
    │   ├── test-cl4whef.json
    │   ├── test-cl4whev.json
    │   ├── test-cl4wxue.json
    │   ├── test-dark-v10.json
    │   ├── test-light-v10.json
    │   ├── test-navigation-guidan.json
    │   ├── test-navigation-guidance.json
    │   ├── test-outdoors-v11.json
    │   ├── test-satellite-st.json
    │   ├── test-satellite-v9.json
    │   └── test-streets-v11.json
    ├── test-data
    │   ├── cl4.json
    │   └── sample1.json
    ├── testresult-20220211.json
    └── testresult-20220211b.json

5 directories, 70 files

maplibre-rs has the maplibre module which contains styles. here we’re trying to parse an input style in json and deserialize it into a rust object.

  • source.rs, style.rs, layer.rs - parsers
  • original directory: original style files
  • tests directory: test data, test results from parser rust tests
  • mod.rs - module layout

serialize

rust object -> json

deserialize

json -> rust object

i’m using the serde library to do this. we made some rust parsers, right now styles, sources, and layers. style includes the layer and source parsers. in the style parser we have a Style and Metadata rust structs. in the source we have a VectorSource struct, in layer we have some funky BackgroundPaint, FillPaint, LinePaint structs part of the enum LayerPaint, and a StyleLayer struct. i think the re-organization will grow organically, and eventually should mirror the style spec: https://maplibre.org/maplibre-gl-js-docs/style-spec/.

wrote some parts of the parser, wrote some tests that input test data max gave me of style exports, good way to flag any attributes/parts of the style spec that weren’t accounted for and iterate on writing the parser that way.

to run tests, in style.rs

cargo test --package maplibre --lib -- style::style::tests

can also add --nocapture or --show-output for verbose flags

cargo test --package maplibre --lib -- style::style::tests::load_sample_data_two --show-output

cargo test --package maplibre --lib -- style::style::tests::load_sample_data_two --show-output > ./maplibre/src/style/tests/testresult-yyymmdd.json

i created a pub extra trait

    #[serde(flatten)]
    pub extra: Map<String, Value>

which uses serde_json::Value parsing any json data using serde_json::from_str function. you don’t have to be explicit, in this case. it grabs everything and dumps it in an object. it’ll be good to pick these apart later and properly deserialize them later, if needed or if they are useful or require some additional computation.

serde has a Map struct which I used https://docs.rs/serde_json/latest/serde_json/struct.Map.html, but maybe we can just use HashMap.

resources