Installation and Getting Started
================================
- Python: 3.7+
- Platforms: Linux, Windows, MacOS
- PyPI package name: `remarking `_
Remarking makes it easy to extract highlights from the ReMarkable cloud.
Install ``remarking``
-------------------------
1. Run the following in your command line:
.. code-block:: text
pip install -U remarking
2. Check the version you have installed
.. code-block:: text
remarking --version
Download your first highlights
------------------------------
Before you download your first highlights you will need to make sure you have actually annotated a document
on your reMarkable.
You can annotate a document by highlighting text on the ReMarkable using the highlight pen.
Once you have some highlighted text you can start extracting it.
.. note::
Remarking requires a one-time authorization token from ``_
in order to access the documents on the reMarkable cloud.
You will be prompted to enter the token if ``remarking`` is not able to authorize with the cloud.
Remarking uses `rmapy `_ to authenticate and access documents on the cloud. ``rmapy`` stores a token in ``~/.rmapi`` so that you do not need to re-authenticate on each invocation of Remarking.
With a document highlighted, you can run ``remarking`` either on the document itself or the folder that contains the document.
Let's say the document is called ``Through the Looking Glass`` and it lives in the ``library`` folder:
.. code-block:: text
> remarking run table library | jq
Extractors: remarkable
Collections: library
Connecting to RM cloud
Connected to RM cloud.
Retrieving cloud metadata
Downloading documents
Downloading "Through the Looking Glass"
Downloaded 1 documents.
Running extractors on documents
Running extractor "RemarkableHighlightExtractor" on "Through the Looking Glass"
Ran extractors and found 1 highlights, 1 are new.
{
"documents": [
{
"id": "2b329909-332d-4c28-b6c1-d298227cc82c",
"version": 5,
"modified_client": 1626736734,
"type": "DocumentType",
"name": "Through the Looking Glass",
"current_page": 11,
"bookmarked": false,
"parent": "d29d67f9-faff-429c-b800-b7815173dcb2"
}
],
"highlights": [
{
"hash": "14235eca5db4a758ad34ab483df737714d10e710a3e3277d726a64da",
"document_id": "2b329909-332d-4c28-b6c1-d298227cc82c",
"text": "Alice was sitting curled up in a corner of the great arm-chair",
"page_number": 11,
"extracted_at": 1626723209,
"extraction_method": "RemarkableHighlightExtractor"
}
]
}
You could also have done:
.. code-block:: text
> remarking run table "Through the Looking Glass"
Extractors: remarkable
Collections: library
Connecting to RM cloud
Connected to RM cloud.
Retrieving cloud metadata
Downloading documents
Downloading "Through the Looking Glass"
Downloaded 1 documents.
Running extractors on documents
Running extractor "RemarkableHighlightExtractor" on "Through the Looking Glass"
highlight_text highlight_page_number document_name
-------------------------------------------------------------- ----------------------- -------------------------
Alice was sitting curled up in a corner of the great arm-chair 11 Through the Looking Glass
.. _structure_of_a_command:
The Structure of a Remarking Command
------------------------------------
All ``remarking`` commands follow a similar structure.
.. code-block:: text
remarking [mode options] [options] args
Modes
*****
The first command is called the mode command. It is one of:
- ``run``
- ``persist``
.. _writer_command:
Writer Commands
***************
The second command specifies which the writer command to use. The built-in writer commands are:
- ``json``
- ``csv``
- ``table``
Learn more about them below in :ref:`writer_commands_deep`.
The documentation section :ref:`writer_command_guide` explains how to add custom writer commands.
Run
---
The ``run`` mode will download all highlights for a given document or folder and output them all according to the
writer you have set.
.. code-block:: text
> remarking run table "Through the Looking Glass"
Extractors: remarkable
Collections: library
Connecting to RM cloud
Connected to RM cloud.
Retrieving cloud metadata
Downloading documents
Downloading "Through the Looking Glass"
Downloaded 1 documents.
Running extractors on documents
Running extractor "RemarkableHighlightExtractor" on "Through the Looking Glass"
highlight_text highlight_page_number document_name
-------------------------------------------------------------- ----------------------- -------------------------
Alice was sitting curled up in a corner of the great arm-chair 11 Through the Looking Glass
The ``run`` mode will always return the same highlights, it does not manage any state.
The ``persist`` run mode, on the other hand, does manage state.
Persist
-------
Remarking shines when you want to keep track of highlights over time.
The ``persist`` mode makes that easy to do.
``persist`` uses the `sqlalchemy `_ ORM to map highlights and documents to any SQLAlchemy supported backing datastore.
SqlAlchemy defaults
*******************
By default, remarking will persist to a local sqlite3 database called ``remarking_database.sqlite3`` that is stored in the current working directory.
Providing an argument for `--sqlalchemy` allows you to use any database that sqlalchemy supports as a backing database.
SqlAlchemy dialects
*******************
Some examples of the `many listed dialects on the sqlalchemy website `_:
- MySQL
- PostgreSQL
- SQLite
.. warning::
You need to install the specific driver for database you would like to connect to.
For example, to connect to a MySQL database, you need to make sure a mysql driver is accessible by
sqlalchemy.
On most systems you will need the ``mysqlclient`` binaries installed on your machine
which can then allow you to install the required python package:
.. code-block:: text
pip install mysqlclient
Please refer to the `sqlalchemy docs for more info `_
An example of connecting to a MySQL database:
.. code-block:: text
> remarking persist --sqlalchemy mysql+pymysql://user:pass@host/dbname?charset=utf8mb4 table library
Extractors: remarkable
Collections: library
Connecting to RM cloud
Connected to RM cloud.
Retrieving cloud metadata
Downloading documents
Downloading "Through the Looking Glass"
Downloaded 1 documents.
Running extractors on documents
Running extractor "RemarkableHighlightExtractor" on "Through the Looking Glass"
Ran extractors and found 1 highlights, 1 are new.
highlight_text highlight_page_number document_name
-------------------------------------------------------------- ----------------------- -------------------------
Alice was sitting curled up in a corner of the great arm-chair 11 Through the Looking Glass
Querying the database will show a ``documents`` and ``highlights`` table that can be queried and futher used.
Another execution of ``persist`` without modifying the document would yield no new highlights.
If the document is modified and a new highlight is made, only the new highlight will be returned on the next run of ``persist``
Setting the ``--sqlalchemy`` option
***************************************
If the argument provided for the ``--sqlalchemy`` is a file, then the first line of that file is read and used as the sqlalchemy connection string. This is useful to avoid leaking secrets.
You can also set the ``REMARKING_SQLALCHEMY`` env var instead of the cmd line option.
.. _extractor_getting_started:
Extractors
----------
Extractors are what actually parse a document and return highlights for it.
You can specify the extractors you want to use via the ``--extractors`` option for both ``run`` and ``persist``.
Listing extractors
******************
You can list all extractors available to use with
.. code-block:: bash
remarking list extractors
Adding new extractors
*********************
Check out :ref:`extractor-guide` for how to create your own extractor.
.. _writer_commands_deep:
Writer Commands
---------------
Writer commands as mentioned in the :ref:`structure_of_a_command` section are the second part of a ``remarking`` command.
Writer commands specify how the highlights should be output.
Viewing Writer Commands
***********************
You can see all available writer commands with:
.. code-block:: bash
remarking run --help
Built-in Writer Commands
************************
Remember that ``run`` will always output all highlights found and that ``persist`` will only output new highlights found since the last execution.
json
****
.. code-block:: text
Usage: remarking run json [OPTIONS] [COLLECTION_NAMES]...
Output highlights and documents as JSON
Options:
-t, --token TEXT One time auth token from for the reMarkable
cloud. Needs only be specified once. [env
var: REMARKING_TOKEN]
-e, --extractors TEXT Comma delimited list of extractors to use.
Run `remarking list extractors` to see valid
extractors. [default: remarkable]
-o, --output FILENAME Output highlights to the given file
-w, --working-directory DIRECTORY
Working directory where files will be
downloaded and highlights generated.
[default: (A randomly generated path within
/tmp/)]
-q, --quiet Print nothing.
-h, --help Show this message and exit.
csv
***
.. code-block:: text
Usage: remarking run csv [OPTIONS] [COLLECTION_NAMES]...
Output highlights normalized with documents as csv.
Check out `remarking list columns` for a list of columns to choose from for
the `--columns` option.
Options:
-t, --token TEXT One time auth token from for the reMarkable
cloud. Needs only be specified once. [env
var: REMARKING_TOKEN]
-e, --extractors TEXT Comma delimited list of extractors to use.
Run `remarking list extractors` to see valid
extractors. [default: remarkable]
-o, --output FILENAME Output highlights to the given file
-w, --working-directory DIRECTORY
Working directory where files will be
downloaded and highlights generated.
[default: (A randomly generated path within
/tmp/)]
-q, --quiet Print nothing.
--delimiter TEXT Delimiter to use to split columns
--columns TEXT Comma delimited list of columns to print
when using plain printing. `remarking list
columns` shows all available columns
[default: highlight_text,document_name,highl
ight_page_number]
-h, --help Show this message and exit.
table
*****
.. code-block:: text
Usage: remarking run table [OPTIONS] [COLLECTION_NAMES]...
Output highlights normalized with documents as a table.
Check out `remarking list columns` for a list of columns to choose from for
the `--columns` option.
Options:
-t, --token TEXT One time auth token from for the reMarkable
cloud. Needs only be specified once. [env
var: REMARKING_TOKEN]
-e, --extractors TEXT Comma delimited list of extractors to use.
Run `remarking list extractors` to see valid
extractors. [default: remarkable]
-o, --output FILENAME Output highlights to the given file
-w, --working-directory DIRECTORY
Working directory where files will be
downloaded and highlights generated.
[default: (A randomly generated path within
/tmp/)]
-q, --quiet Print nothing.
--truncate / --no-truncate Truncate results when printing plain
--plain / --no-plain Output one data entry per line.
--columns TEXT Comma delimited list of columns to print
when using plain printing. `remarking list
columns` shows all available columns
[default: highlight_text,document_name,highl
ight_page_number]
-h, --help Show this message and exit.