ripgrep-all/README.md

76 lines
3.0 KiB
Markdown
Raw Normal View History

2019-06-12 19:29:56 +00:00
# rga - ripgrep, but also search in PDFs, E-Books, Office documents, zip, tar.gz, etc
2019-06-13 14:06:43 +00:00
rga is a line-oriented search tool that allows you to look for a regex in a multitude of file types. It is a wrapper around the awesome [ripgrep] that enables it to search in pdf, docx, pptx, movie subtitles (mkv, mp4), sqlite, etc.
2019-06-12 20:32:20 +00:00
2019-06-12 21:06:50 +00:00
[![Linux build status](https://api.travis-ci.org/phiresky/ripgrep_all.svg)](https://travis-ci.org/phiresky/ripgrep_all)
2019-06-12 20:06:21 +00:00
[![Crates.io](https://img.shields.io/crates/v/ripgrep_all.svg)](https://crates.io/crates/ripgrep_all)
2019-06-12 19:55:42 +00:00
2019-06-13 14:06:43 +00:00
## Future Work
2019-06-06 21:19:59 +00:00
2019-06-13 14:06:43 +00:00
- I wanted to add a photograph adapter (based on object classification / detection) for fun, based on something . It worked with [YOLO](https://pjreddie.com/darknet/yolo/), but something more useful and state-of-the art [like this](https://github.com/aimagelab/show-control-and-tell) proved very hard to integrate.
- 7z adapter (couldn't find a nice to use Rust library)
- allow per-adapter configuration options (probably via env (RGA_ADAPTER_CONF=json))
2019-06-06 21:19:59 +00:00
2019-06-13 14:06:43 +00:00
## Examples
2019-06-06 15:59:15 +00:00
2019-06-13 14:06:43 +00:00
Say you have a large folder of papers or lecture slides, and you can't remember which one of them mentioned `LSTM`s. With rga, you can just run this:
2019-06-07 19:46:03 +00:00
2019-06-13 14:06:43 +00:00
```
rga "LSTM|GRU" collection/
[results]
```
and it will recursively find a regex in pdfs and pptx slides, including if some of them are zipped up.
You can do mostly the same thing with [`pdfgrep -r`][pdfgrep], but it will be much slower and you will miss content in other file types.
```barchart
title: Searching in 20 pdfs with 100 slides each
subtitle: lower is better
data:
- pdfgrep: 123s
- rga (first run): 10.3s
- rga (subsequent runs): 0.1s
```
2019-06-12 19:37:15 +00:00
2019-06-13 14:06:43 +00:00
On the first run rga is mostly faster because of multithreading, but on subsequent runs (on the same files but with any query) rga will cache the text extraction because pdf parsing is slow.
## Setup
rga should compile with stable Rust. To install it, simply run (your OSes equivalent of)
2019-06-12 19:55:42 +00:00
```bash
2019-06-12 20:11:20 +00:00
apt install build-essential pandoc poppler-utils
2019-06-12 19:55:42 +00:00
cargo install ripgrep_all
rga --help # works! :)
2019-06-12 19:55:42 +00:00
```
2019-06-13 14:06:43 +00:00
## Technical details
`rga` simply runs ripgrep (`rg`) with some options set, especially `--pre=rga-preproc` and `--pre-glob`.
`rga-preproc [fname]` will match an adapter to the given file based on either it's filename or it's mime type (if `--accurate` is given).
2019-06-12 19:37:15 +00:00
Some rga adapters run external binaries
2019-06-13 14:06:43 +00:00
## Development
2019-06-07 19:46:03 +00:00
To enable debug logging:
2019-06-07 21:04:18 +00:00
```bash
2019-06-12 15:23:30 +00:00
export RUST_LOG=debug
2019-06-07 19:46:03 +00:00
export RUST_BACKTRACE=1
2019-06-07 21:04:18 +00:00
```
2019-06-11 18:35:20 +00:00
Also rember to disable caching with `--rga-no-cache` or clear the cache in `~/.cache/rga` to debug the adapters.
2019-06-12 20:32:20 +00:00
# Similar tools
2019-06-13 14:06:43 +00:00
- [pdfgrep][pdfgrep]
2019-06-12 20:32:20 +00:00
- [this gist](https://gist.github.com/phiresky/5025490526ba70663ab3b8af6c40a8db) has my proof of concept version of a caching extractor to use ripgrep as a replacement for pdfgrep.
- [this gist](https://gist.github.com/ColonolBuendia/314826e37ec35c616d70506c38dc65aa) is a more extensive preprocessing script by [@ColonolBuendia](https://github.com/ColonolBuendia)
2019-06-13 14:06:43 +00:00
[pdfgrep]: https://pdfgrep.org/
[ripgrep]: https://github.com/BurntSushi/ripgrep