Runner script and extended docs #5

tobiaobr · 2025-11-03T14:54:09Z

Added Script that runs the PotreeConverter on all H3 cells
Updated Readme to reflect these changes
Added PotreeConverter Package to the repo

It needs to be initialized before it can be ran, readme describes it
Used this fork ensure macOS compatibility

Summary by CodeRabbit

New Features
- End-to-end point‑cloud conversion pipeline: chunking, spatial refinement, octree hierarchy building and indexing
- Faster conversions via multi‑threaded processing and concurrent batched writing
- Multiple sampling strategies (random, Poisson variants) and runtime progress/monitoring
- Debug/export utilities for inspecting node and attribute data; Brotli compression support and native build integration
Chores
- Added build/config files, editor workspace settings, and expanded ignore rules

coderabbit · 2025-11-03T14:54:31Z

Walkthrough

Adds a PotreeConverter project with C++20 build, point-cloud chunking/refinement, octree hierarchy construction, multiple samplers, concurrent/writer and monitor utilities, LAS attribute handling, debug exporters, and integrates a complete Brotli compression subtree (encoder, decoder, build scripts, and support libraries).

Changes

Cohort / File(s)	Summary
Repo & Editor `/.gitignore`, `PotreeConverter/.gitignore`, `PotreeConverter/.vscode/*`	Update ignore rules (preserve `/.venv`, add `pointclouds`, `PotreeConverter/build`, `conversion_log.txt`); add VSCode settings, keybindings, and launch config for PotreeConverter.
Top-level CMake & Executable `PotreeConverter/CMakeLists.txt`	Add CMake for PotreeConverter (C++20), define PotreeConverter executable, add laszip/brotli subprojects, thread/TBB handling on UNIX, and post-build resource/license copying.
Brotli subtree & packaging `PotreeConverter/Converter/libs/brotli/*`	Add full Brotli sources, headers, build files (CMake, Makefile, Autotools, bootstrap, WORKSPACE, MANIFEST), docs (README/CONTRIBUTING), LICENSE, and many encoder/decoder C modules and headers.
Brotli common/data tables `.../c/common/*`	Add constants, context lookup table, transform API, dictionary API, platform abstractions, default alloc/free, version header, and static data arrays.
Brotli decoder `.../c/dec/*`	Add bit-reader API/impl, Huffman builders, command LUT, decoder state machinery, and full decoder implementation with streaming APIs.
Brotli encoder core `.../c/enc/`, `command.`, `brotli_bit_stream.*`	Add encoder entrypoints, bitstream assembly, command tables, fragment compressors (fast & two-pass), and related headers.
Brotli encoder advanced `.../c/enc/backward_references`, `cluster`, `block_splitter`, `bit_cost`	Add backward-reference generators (including Zopfli/HQ), clustering and histogram utilities, block-splitting, bit-cost estimators, dictionary hash, and encoder-dictionary initializer.
Arguments utility `PotreeConverter/Converter/libs/arguments/Arguments.hpp`	Add a small command-line parsing utility (AValue, Argument, Arguments) with typed accessors and usage generation.
Core types & utilities `PotreeConverter/Converter/include/Vector3.h`, `Attributes.h`, `converter_utils.h`	Add Vector3 math type; AttributeType/Attribute/Attributes system and helpers; core structures (Source, State, BoundingBox, Options), Morton helpers, and misc utilities.
Structures & Samplers `PotreeConverter/Converter/include/structures.h`, `sampler_*.h`	Add Node tree, CumulativeColor, Sampler interface and SamplerState; implement SamplerRandom, SamplerPoisson, SamplerPoissonAverage sampling strategies.
Chunking, refinement & hierarchy `PotreeConverter/Converter/include/ChunkRefiner.h`, `HierarchyBuilder.h`, `indexer.h`, `chunker_countsort_laszip.h`	Add chunk discovery/refinement, per-chunk Morton-grid subdivision and parallel counting, hierarchy batch loading/serialization, octree indexing and flusher, and chunking API declarations.
I/O, concurrency & monitoring `PotreeConverter/Converter/include/ConcurrentWriter.h`, `Monitor.h`, `logger.h`, `DbgWriter.h`, `PotreeConverter.h`	Add multi-threaded per-path buffered writer, runtime Monitor, lightweight logger macros, debug exporters for node data, LAS attribute parsing, and computeScaleOffset / attribute aggregation utilities.
Misc & small fixes `PotreeConverter/Converter/include/prototyping.h`	Add missing `#pragma once` and minor header scaffolding.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant Main as main.cpp
    participant Chunker as Chunker
    participant Refiner as ChunkRefiner
    participant Indexer as Indexer
    participant Writer as ConcurrentWriter
    participant Monitor as Monitor

    Main->>Chunker: doChunking(sources, targetDir, bounds)
    Chunker->>Chunker: read headers and write chunk files
    Chunker->>Refiner: refine(targetDir, state)
    Refiner->>Refiner: getChunks(pathIn) -> select large chunks
    Refiner->>Refiner: refineChunk() (parallel Morton grid counting)
    Refiner->>Indexer: submit refined chunkParts
    Indexer->>Indexer: build octree nodes & hierarchy (async flush)
    Indexer->>Writer: writeAndUnload(node)
    Writer->>Writer: batch & flush files to disk (multi-threaded)
    Monitor->>Main: periodic status updates (progress, throughput, RAM)

sequenceDiagram
    autonumber
    participant Stream as CompressedInput
    participant Decoder as BrotliDecoder
    participant BitReader as BrotliBitReader
    participant Huffman as HuffmanBuilder
    participant Output as RingBuffer

    Stream->>Decoder: BrotliDecoderDecompressStream()
    Decoder->>BitReader: BrotliInitBitReader / Warmup
    BitReader->>BitReader: fill bit window
    Decoder->>Huffman: build/load Huffman tables for metablock
    Huffman->>Decoder: provide decode symbols
    Decoder->>Output: emit literals & copies into ring buffer
    Decoder->>Decoder: repeat per metablock until finished

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120+ minutes

Areas needing focused review:

Brotli encoder/decoder internals: bit-level logic, Huffman table builders, state machines, memory management, and correctness.
Large C template/inc patterns and generated variants (backward_references*, cluster_inc, block_splitter_inc, bit_cost_inc).
Concurrency and lifecycle: ConcurrentWriter, Indexer writer/backlog, HierarchyFlusher, and Monitor thread handling.
Spatial algorithms and parallel processing: ChunkRefiner (Morton encoding, grid counting) and HierarchyBuilder serialization.
Sampler implementations: correctness of sampling, buffer ownership, and multithreaded sorts.
LAS attribute parsing, computeOutputAttributes aggregation across multiple headers.
Build integration: CMake brotli/laszip integration, platform options, and post-build resource copying.

Poem

"I hopped through bytes and clusters bright,
nibbling extra attributes by moonlight.
I mixed Morton codes with crunchy crunch,
braided Brotli bits for every bunch.
— a joyful rabbit with a converter's delight 🐇"

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Free

📥 Commits

Reviewing files that changed from the base of the PR and between cd7db5f and 1f4b171.

📒 Files selected for processing (1)

PotreeConverter/CMakeLists.txt (1 hunks)

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

adriahso

I get the following error when trying to run ../cmake:

CMake Error at Converter/libs/brotli/CMakeLists.txt:5 (cmake_minimum_required):
  Compatibility with CMake < 3.5 has been removed from CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.

  Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.

I remember previously having to change cmake_minimum_required(VERSION 2.8.6)to cmake_minimum_required(VERSION 3.16) in PotreeConverter/Converter/libs/brotli/CMakeLists.txt when running PotreeConverter previously, but when I do now I get:

-- Build type is 'Release'
-- Compiler is not EMSCRIPTEN
CMake Warning at Converter/libs/brotli/CMakeLists.txt:319 (message):
  Test file tests/testdata/alice29.txt does not exist.


CMake Warning at Converter/libs/brotli/CMakeLists.txt:319 (message):
  Test file tests/testdata/asyoulik.txt does not exist.


CMake Warning at Converter/libs/brotli/CMakeLists.txt:319 (message):
  Test file tests/testdata/lcet10.txt does not exist.


CMake Warning at Converter/libs/brotli/CMakeLists.txt:319 (message):
  Test file tests/testdata/plrabn12.txt does not exist.


CMake Error at CMakeLists.txt:58 (find_package):
  By not providing "FindTBB.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "TBB", but
  CMake did not find one.

  Could not find a package configuration file provided by "TBB" with any of
  the following names:

    TBBConfig.cmake
    tbb-config.cmake

  Add the installation prefix of "TBB" to CMAKE_PREFIX_PATH or set "TBB_DIR"
  to a directory containing one of the above files.  If "TBB" provides a
  separate development package or SDK, be sure it has been installed.

README.MD

Co-authored-by: Adrian Haabpiht Solberg <adriahso@stud.ntnu.no>

gautegf · 2025-11-09T17:34:16Z

I get the following error when trying to run ../cmake:

CMake Error at Converter/libs/brotli/CMakeLists.txt:5 (cmake_minimum_required):
  Compatibility with CMake < 3.5 has been removed from CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.

  Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.

I remember previously having to change cmake_minimum_required(VERSION 2.8.6)to cmake_minimum_required(VERSION 3.16) in PotreeConverter/Converter/libs/brotli/CMakeLists.txt when running PotreeConverter previously, but when I do now I get:

-- Build type is 'Release'
-- Compiler is not EMSCRIPTEN
CMake Warning at Converter/libs/brotli/CMakeLists.txt:319 (message):
  Test file tests/testdata/alice29.txt does not exist.


CMake Warning at Converter/libs/brotli/CMakeLists.txt:319 (message):
  Test file tests/testdata/asyoulik.txt does not exist.


CMake Warning at Converter/libs/brotli/CMakeLists.txt:319 (message):
  Test file tests/testdata/lcet10.txt does not exist.


CMake Warning at Converter/libs/brotli/CMakeLists.txt:319 (message):
  Test file tests/testdata/plrabn12.txt does not exist.


CMake Error at CMakeLists.txt:58 (find_package):
  By not providing "FindTBB.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "TBB", but
  CMake did not find one.

  Could not find a package configuration file provided by "TBB" with any of
  the following names:

    TBBConfig.cmake
    tbb-config.cmake

  Add the installation prefix of "TBB" to CMAKE_PREFIX_PATH or set "TBB_DIR"
  to a directory containing one of the above files.  If "TBB" provides a
  separate development package or SDK, be sure it has been installed.

I get the exact same issue, but me and Adrian both have mac, could be something to investigate

…ility

tobiaobr · 2025-11-10T07:10:37Z

I get the following error when trying to run ../cmake:

CMake Error at Converter/libs/brotli/CMakeLists.txt:5 (cmake_minimum_required):
  Compatibility with CMake < 3.5 has been removed from CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.

  Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.

I remember previously having to change cmake_minimum_required(VERSION 2.8.6)to cmake_minimum_required(VERSION 3.16) in PotreeConverter/Converter/libs/brotli/CMakeLists.txt when running PotreeConverter previously, but when I do now I get:

-- Build type is 'Release'
-- Compiler is not EMSCRIPTEN
CMake Warning at Converter/libs/brotli/CMakeLists.txt:319 (message):
  Test file tests/testdata/alice29.txt does not exist.


CMake Warning at Converter/libs/brotli/CMakeLists.txt:319 (message):
  Test file tests/testdata/asyoulik.txt does not exist.


CMake Warning at Converter/libs/brotli/CMakeLists.txt:319 (message):
  Test file tests/testdata/lcet10.txt does not exist.


CMake Warning at Converter/libs/brotli/CMakeLists.txt:319 (message):
  Test file tests/testdata/plrabn12.txt does not exist.


CMake Error at CMakeLists.txt:58 (find_package):
  By not providing "FindTBB.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "TBB", but
  CMake did not find one.

  Could not find a package configuration file provided by "TBB" with any of
  the following names:

    TBBConfig.cmake
    tbb-config.cmake

  Add the installation prefix of "TBB" to CMAKE_PREFIX_PATH or set "TBB_DIR"
  to a directory containing one of the above files.  If "TBB" provides a
  separate development package or SDK, be sure it has been installed.

It should be resolved with the latest commit

adriahso

Nice, works well now!

gautegf

LGTM

tobiaobr added 4 commits November 3, 2025 13:33

Remove existing PotreeConverter from index to add as submodule

7a1f34f

removed submodule

268d515

Remove PotreeConverter as submodule but keep folder locally

7b942bb

added Potreeconverter build as subdirectory

5fbbebb

tobiaobr self-assigned this Nov 3, 2025

tobiaobr requested a review from adriahso November 3, 2025 14:55

updated docs

b8500e0

tobiaobr force-pushed the runner-script-and-extended-docs branch from 33301a3 to b8500e0 Compare November 3, 2025 15:01

adriahso requested changes Nov 7, 2025

View reviewed changes

README.MD Outdated Show resolved Hide resolved

README.MD Outdated Show resolved Hide resolved

README.MD Show resolved Hide resolved

tobiaobr and others added 3 commits November 8, 2025 15:10

Update README.MD

e6aecdb

Co-authored-by: Adrian Haabpiht Solberg <adriahso@stud.ntnu.no>

Update README.MD

6412703

Co-authored-by: Adrian Haabpiht Solberg <adriahso@stud.ntnu.no>

fixed readme typos & added link to potreeconverter docs

b6526ce

tobiaobr added 2 commits November 9, 2025 21:35

changed cmake version requirement

48fbac9

Changed PotreeConverter to use LeftofHere fork to ensure mac compatib…

4c159ed

…ility

tobiaobr force-pushed the runner-script-and-extended-docs branch from 1fc2892 to 4c159ed Compare November 10, 2025 07:07

tobiaobr added 3 commits November 10, 2025 08:26

fixed cmake version config file

cd7db5f

yet another version fix

1f4b171

README finishing touches

054fd4c

tobiaobr force-pushed the runner-script-and-extended-docs branch from 81f9e2b to 054fd4c Compare November 10, 2025 07:40

adriahso approved these changes Nov 10, 2025

View reviewed changes

gautegf approved these changes Nov 10, 2025

View reviewed changes

tobiaobr merged commit ecc9b7e into main Nov 10, 2025
1 check passed

tobiaobr deleted the runner-script-and-extended-docs branch November 10, 2025 07:47

Runner script and extended docs #5

Runner script and extended docs #5

tobiaobr commented Nov 3, 2025 •

edited by coderabbit bot

Loading

coderabbit bot commented Nov 3, 2025 •

edited

Loading

adriahso left a comment

gautegf commented Nov 9, 2025

tobiaobr commented Nov 10, 2025

adriahso left a comment

gautegf left a comment

Runner script and extended docs #5

Runner script and extended docs #5

Conversation

tobiaobr commented Nov 3, 2025 • edited by coderabbit bot Loading

Summary by CodeRabbit

coderabbit bot commented Nov 3, 2025 • edited Loading

Walkthrough

Changes

Sequence Diagram(s)

Estimated code review effort

Poem

adriahso left a comment

Choose a reason for hiding this comment

gautegf commented Nov 9, 2025

tobiaobr commented Nov 10, 2025

adriahso left a comment

Choose a reason for hiding this comment

gautegf left a comment

Choose a reason for hiding this comment

tobiaobr commented Nov 3, 2025 •

edited by coderabbit bot

Loading

coderabbit bot commented Nov 3, 2025 •

edited

Loading