Handwriting Recognition with Markdowns!
I was in training recently for senior technical leaders. This post is about the interesting giveaway that we received, Rocketbooks.
- It’s a fully reusable notebook.
- The paper quality is made for Pilot Frixion pens, which can be wiped clean with no damages.
- It comes with a Rocketbook App that scans and transcribes your notes and sends it to your desired destination. (Email, Evernote, Onedrive, etc.)
What a fun giveaway! I love using it and testing its limits. As I was exploring, I thought of extending a recent POC because of this suggestion to support markdowns. Wouldn’t it be cool if we can convert the following notes to rich text?
This post is about how I extended our original handwriting recognition POC to support Markdowns.
Conceptual Flow
- Have two triggers: New Emails and New OneDrive files
- Move the core logic into an HTTP triggered logic app and send a message output to a Service Bus Topic
- Send the output to Teams and Sharepoint like before as Service Bus Subscriptions, but also add OneNote where markdown is supported.
Actual Implementation
Core OCR Logic App
The logic is mostly the same as the original; you may check the details in my last post. But here are the key differences:
- Starts with an HTTP Request (accepting a JSON body) and responds with an HTTP 200 upon completion
- Instead of using a Key Vault connector, I used an HTTP GET to authenticate using the logic app managed identity (currently in preview). I learned this approach from here.
- Send the OCR output to a Service Bus Topic as a JSON message. Make sure that JSON is valid by escaping the OCR output characters via a Compose action with the following expression:
1
replace(replace(body('ParseRecognizeTextJsonResponse'),'\','\\'),'"','\"')
Triggers
Email Flow Trigger
Just for fun, I decided to use Microsoft Power Automate (previously known as Flow) for my e-mail trigger instead of Azure Logic Apps.
Pro: My Power Automate will trigger immediately upon receiving an e-mail. Whereby using Azure Logic Apps will only trigger by a specified schedule.
Con: No Visual Studio support. I can’t easily get the code and save it into my GIT repo using the VS Cloud Explorer. (Though there is a long workaround involving Peak Code and manual saving.)
OneDrive (Personal) Logic App Trigger
I used Azure Logic Apps for this trigger due to the following limitations:
- RocketBook does not support OneDrive for Business as a save destination
- My company policies do not allow the use of OneDrive (personal) with PowerAutomate.
Input
SharePoint and Teams
Other than reading from the service bus, the SharePoint and Teams implementations were simply extracted from my last logic app.
Change Notes:
The ServiceBus message is base64 encoded, so I used this expression before parsing:
@base64ToString(triggerBody()?['ContentData'])in the Parse JSON action.The MS Teams adaptive card JSON must be valid. I used this expression to create a JSON escaped string value:
@{replace(replace(body('Parse_JSON')?['Text'],'\\','\\\\'),'\"','\\\"')}
OneNote
The OneNote implementation is almost the same; the hard part is creating a function to process the markdown.
Converting the Markdown Text to OneNote HTML
To create the OneNote HTML, I referred to this documentation and created the following Markdown to HTML mapping:
| Markdown | OneNote HTML |
|---|---|
| #This is a heading# | <h1>This is a heading<\/h1> |
| *This is in bold* | <b>This is in bold<\/b> |
| !This is in italic! | <i>This is in italic<\/i> |
| 1. This is 2 an 3 . ordered list | <ol> <li>This is<\/li> <li>an<\/li> <li>ordered list<\/li> <\/ol> |
| > This is > an > unordered list | <ul> <li>This is<\/li> <li>an<\/li> <li>unordered list<\/li> <\/ul> |
And expect the output HTML template to be something like this
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
<html>
<head>
<title>@{body('Parse_JSON')?['Name']}</title>
<meta name='created' content='@{body('Parse_JSON')?['CreatedDateTime']}' />
</head>
<body>
<img src='@{body('Parse_JSON')?['ImageURL']}' alt='@{body('Parse_JSON')?['Name']}' />
<h1>heading1</h1>
<b>This is bold</b>
<i>This is italic</i>
<ul>
<li>bullet1</li>
<li>bullet2</li>
<li>bullet3</li>
</ul>
<ol>
<li>number1</li>
<li>number2</li>
<li>number3</li>
</ol>
</body>
</html>
The rest is to code the Azure function, which I shared here.
Creating a Page in SharePoint
Using the Logic App designer view will cause issues because it uses a rich text box for the body content (as seen in the screenshot above). This will add <p>BODY</p> to the content automatically.
To avert this, I edited the code in the Logic App Code View and changed the body value appropriately.
1
2
3
4
5
6
7
8
9
10
11
"Create_page_in_a_section": {
"runAfter": {
"ConvertMarkdownTextToOneNoteHtml": [
"Succeeded"
]
},
"type": "ApiConnection",
"inputs": {
"body": "@{body('ConvertMarkdownTextToOneNoteHtml')}",
...
Output
You may check the source code here (Note: Code doesn’t include the Power Automate e-mail trigger).
In case you don’t have a Rocketbook, here are some excellent alternatives:
- Use Microsoft Office Lens (Android or iOS), which can quickly scan your handwritten notes and upload them to a configured OneDrive folder.
- Use the scan feature of the OneDrive Mobile App
But this will work even with a regular photo of your whiteboard or notebook.