Repos as first-objects
Brock Lampman
I have a simple and effective approach to turn several repos into a single parquet that AI does better at reasoning over with SQL than grep. It involves a CSV manifest of 'allow_patterns' since repos sometimes contain significant noise (could be made nicer in MD UI). The resultant parquet/dataset has columns: repo, path, filename, ext, text, path_pattern. This would work with ssh-key secrets in MD and draw from default branches. I also do something similar with commits.
I feel this is straightforward-enough of an approach on a high-ROI benefit to many MD users.
To add, I also use Git as a reverse-flow data-versioning mechanism where right-sized and right-churn-rate datasets are materialized as ordered-CSV and push to Git. Not sure this is as compelling, especially given MDs snapshot-time travel-retention features, but since it's on the same topic (repos as first-objects), thought it worth mentioning. Reach out for any questions.