etc/sfm-js
1
0
Fork 0
mirror of https://activitypub.software/TransFem-org/sfm-js synced 2024-11-23 06:25:14 +00:00
Code Issues Projects Releases Packages Wiki Activity

TypeScript版パーサーのマージ (#124)

Browse source 
* implement parser with TypeScript (#116)

* clean parser

* parser, success, failure, str, parser.map

* seq

* atLeast, any, alt, match, notMatch

* mergeText

* improve seq

* lazy, createLanguage

* types

* regexp, refactor

* nest limit

* lint

* state

* syntaxes

* sep1, succeeded, option, fn

* simple

* strikeWave, plainTag, inlineCode, mathInline

* mention, refactor

* seqPartial

* 🚀

* parser trace

* fix mention, implement hashtag

* lineBegin, lineEnd, refactor

* imple codeBlock, fix lineEnd

* codeBlock, mathBlock

* fix codeBlock

* fix mathBlock

* fix codeBlock

* lint

* fix inlineCode

* 🚀

* centerTag

* fix nesting limit

* fix unicodeEmoji

* 🚀

* search

* refactor

* seqPartial -> seqOrText

* lint

* url, urlAlt

* 🚀

* 🚀

* text

* fix

* link

* linkLabel state

* lint

* nesting limit for link label

* fix url bracket pair

* nest

* refactor

* refactor

* remove

* add test

* wip quote

* add quote test

* quote

* refactor

* hashtag

* refactor

* type

* type

* refactor

* lint

* url

* italicAsta, italicUnder

* italicAsta, italicUnder, mention, rethink spec

* rethink spec

* test: change implementation-dependent parts

* hashtag

* add mention test

* mention

* mention

* mention

* mention

* url

* test

* hashtag

* Revert "Auxiliary commit to revert individual files from 373972beef10eb99ff3e3635a32a207854154a2a"

This reverts commit 622b66e20778ad5c283ea7629db853cbf2bb601f.

* package-lock

* Update tsconfig.json

* Update tsconfig.json

* ignore a tsd error when importing twemoji-parser regexp

* lint

* lint

Co-authored-by: syuilo <Syuilotan@yahoo.co.jp>

* v0.23.0-canary.1

* readme

* update chagelog

* update changelog

* update changelog

* refactor

* update core combinators

Co-authored-by: syuilo <Syuilotan@yahoo.co.jp>
This commit is contained in:
marihachi 2022-07-22 02:21:56 +09:00 committed by GitHub
parent 7b7af907bd
commit 5fe291a7e7
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
14 changed files with 1179 additions and 1018 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
CHANGELOG.md
View file

@ -11,6 +11,20 @@
--> -->
## 0.23.0 (unreleased)
### Features
- Add Plain syntax (#101)
### Improvements
- The parser is now implemented in TypeScript! 🎉 (#92)
- Disable all syntax when nesting limited (#90)
### Changes
- Rename existing plain series (#113):
- parsePlain -> parseSimple
- MfmPlainNode -> MfmSimpleNode
## 0.22.1 ## 0.22.1
npm: https://www.npmjs.com/package/mfm-js/v/0.22.1 npm: https://www.npmjs.com/package/mfm-js/v/0.22.1

2
README.md
View file

@ -1,5 +1,5 @@
# mfm.js # mfm.js
An MFM parser implementation with PEG.js. An MFM parser implementation with TypeScript.
[Try it out!](https://runkit.com/npm/mfm-js) [Try it out!](https://runkit.com/npm/mfm-js)
[![Test](https://github.com/misskey-dev/mfm.js/actions/workflows/test.yml/badge.svg)](https://github.com/misskey-dev/mfm.js/actions/workflows/test.yml) [![Test](https://github.com/misskey-dev/mfm.js/actions/workflows/test.yml/badge.svg)](https://github.com/misskey-dev/mfm.js/actions/workflows/test.yml)

6
docs/syntax.md
View file

@ -295,7 +295,7 @@ _italic_
構文2,3のみ: 構文2,3のみ:
※1つ目の`*`と`_`を開始記号と呼ぶ。 ※1つ目の`*`と`_`を開始記号と呼ぶ。
- 内容には`[a-z0-9 \t]i`にマッチする文字が使用できる。 - 内容には`[a-z0-9 \t]i`にマッチする文字が使用できる。
- 開始記号の前の文字が(無い、改行、半角スペース、[a-zA-Z0-9]に一致しない)のいずれかの時にイタリック文字として判定される。 - 開始記号の前の文字が`[a-z0-9]i`に一致しない時にイタリック文字として判定される。
## ノード ## ノード
```js ```js
@ -402,7 +402,7 @@ _italic_
``` ```
## 詳細 ## 詳細
- 最初の`@`の前の文字が(改行、スペース、無し、[a-zA-Z0-9]に一致しない)のいずれかの場合にメンションとして認識する。 - 最初の`@`の前の文字が`[a-z0-9]i`に一致しない場合にメンションとして認識する。
### ユーザ名 ### ユーザ名
- 1文字以上。 - 1文字以上。
@ -451,7 +451,7 @@ _italic_
- 内容には半角スペース、全角スペース、改行、タブ文字を含めることができない。 - 内容には半角スペース、全角スペース、改行、タブ文字を含めることができない。
- 内容には`.` `,` `!` `?` `'` `"` `#` `:` `/` `【` `】` `<` `>` `【` `】` `(` `)` `「` `」` `（` `）` を含めることができない。 - 内容には`.` `,` `!` `?` `'` `"` `#` `:` `/` `【` `】` `<` `>` `【` `】` `(` `)` `「` `」` `（` `）` を含めることができない。
- 括弧は対になっている時のみ内容に含めることができる。対象: `()` `[]` `「」` `（）` - 括弧は対になっている時のみ内容に含めることができる。対象: `()` `[]` `「」` `（）`
- `#`の前の文字が(改行、スペース、無し、[a-zA-Z0-9]に一致しない)のいずれかの場合にハッシュタグとして認識する。 - `#`の前の文字が`[a-z0-9]i`に一致しない場合にハッシュタグとして認識する。
- 内容が数字のみの場合はハッシュタグとして認識しない。 - 内容が数字のみの場合はハッシュタグとして認識しない。
## ノード ## ノード

321
package-lock.json generated
View file

@ -17,10 +17,8 @@
"@types/node": "18.0.3", "@types/node": "18.0.3",
"@typescript-eslint/eslint-plugin": "^5.30.5", "@typescript-eslint/eslint-plugin": "^5.30.5",
"@typescript-eslint/parser": "^5.30.5", "@typescript-eslint/parser": "^5.30.5",
"copyfiles": "^2.4.1",
"eslint": "^8.19.0", "eslint": "^8.19.0",
"jest": "^28.1.2", "jest": "^28.1.2",
"peggy": "1.2.0",
"ts-jest": "^28.0.5", "ts-jest": "^28.0.5",
"ts-node": "10.8.2", "ts-node": "10.8.2",
"tsd": "^0.22.0", "tsd": "^0.22.0",
@ -2116,31 +2114,6 @@
"safe-buffer": "~5.1.1" "safe-buffer": "~5.1.1"
} }
}, },
"node_modules/copyfiles": {
"version": "2.4.1",
"resolved": "https://registry.npmjs.org/copyfiles/-/copyfiles-2.4.1.tgz",
"integrity": "sha512-fereAvAvxDrQDOXybk3Qu3dPbOoKoysFMWtkY3mv5BsL8//OSZVL5DCLYqgRfY5cWirgRzlC+WSrxp6Bo3eNZg==",
"dev": true,
"dependencies": {
"glob": "^7.0.5",
"minimatch": "^3.0.3",
"mkdirp": "^1.0.4",
"noms": "0.0.0",
"through2": "^2.0.1",
"untildify": "^4.0.0",
"yargs": "^16.1.0"
},
"bin": {
"copyfiles": "copyfiles",
"copyup": "copyfiles"
}
},
"node_modules/core-util-is": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.2.tgz",
"integrity": "sha1-tf1UIgqivFq1eqtxQMlAdUUDwac=",
"dev": true
},
"node_modules/create-require": { "node_modules/create-require": {
"version": "1.1.1", "version": "1.1.1",
"resolved": "https://registry.npmjs.org/create-require/-/create-require-1.1.1.tgz", "resolved": "https://registry.npmjs.org/create-require/-/create-require-1.1.1.tgz",
@ -3151,12 +3124,6 @@
"url": "https://github.com/sponsors/sindresorhus" "url": "https://github.com/sponsors/sindresorhus"
} }
}, },
"node_modules/isarray": {
"version": "0.0.1",
"resolved": "https://registry.npmjs.org/isarray/-/isarray-0.0.1.tgz",
"integrity": "sha1-ihis/Kmo9Bd+Cav8YDiTmwXR7t8=",
"dev": true
},
"node_modules/isexe": { "node_modules/isexe": {
"version": "2.0.0", "version": "2.0.0",
"resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz",
@ -4178,18 +4145,6 @@
"node": ">= 6" "node": ">= 6"
} }
}, },
"node_modules/mkdirp": {
"version": "1.0.4",
"resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-1.0.4.tgz",
"integrity": "sha512-vVqVZQyf3WLx2Shd0qJ9xuvqgAyKPLAiqITEtqW0oIUjzo3PePDd6fW9iFz30ef7Ysp/oiWqbhszeGWW2T6Gzw==",
"dev": true,
"bin": {
"mkdirp": "bin/cmd.js"
},
"engines": {
"node": ">=10"
}
},
"node_modules/ms": { "node_modules/ms": {
"version": "2.1.2", "version": "2.1.2",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.2.tgz", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.2.tgz",
@ -4214,16 +4169,6 @@
"integrity": "sha512-PiVXnNuFm5+iYkLBNeq5211hvO38y63T0i2KKh2KnUs3RpzJ+JtODFjkD8yjLwnDkTYF1eKXheUwdssR+NRZdg==", "integrity": "sha512-PiVXnNuFm5+iYkLBNeq5211hvO38y63T0i2KKh2KnUs3RpzJ+JtODFjkD8yjLwnDkTYF1eKXheUwdssR+NRZdg==",
"dev": true "dev": true
}, },
"node_modules/noms": {
"version": "0.0.0",
"resolved": "https://registry.npmjs.org/noms/-/noms-0.0.0.tgz",
"integrity": "sha1-2o69nzr51nYJGbJ9nNyAkqczKFk=",
"dev": true,
"dependencies": {
"inherits": "^2.0.1",
"readable-stream": "~1.0.31"
}
},
"node_modules/normalize-package-data": { "node_modules/normalize-package-data": {
"version": "3.0.2", "version": "3.0.2",
"resolved": "https://registry.npmjs.org/normalize-package-data/-/normalize-package-data-3.0.2.tgz", "resolved": "https://registry.npmjs.org/normalize-package-data/-/normalize-package-data-3.0.2.tgz",
@ -4409,18 +4354,6 @@
"node": ">=8" "node": ">=8"
} }
}, },
"node_modules/peggy": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/peggy/-/peggy-1.2.0.tgz",
"integrity": "sha512-PQ+NKpAobImfMprYQtc4Egmyi29bidRGEX0kKjCU5uuW09s0Cthwqhfy7mLkwcB4VcgacE5L/ZjruD/kOPCUUw==",
"dev": true,
"bin": {
"peggy": "bin/peggy"
},
"engines": {
"node": ">=10"
}
},
"node_modules/picocolors": { "node_modules/picocolors": {
"version": "1.0.0", "version": "1.0.0",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.0.0.tgz", "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.0.0.tgz",
@ -4511,12 +4444,6 @@
"url": "https://github.com/chalk/ansi-styles?sponsor=1" "url": "https://github.com/chalk/ansi-styles?sponsor=1"
} }
}, },
"node_modules/process-nextick-args": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-2.0.1.tgz",
"integrity": "sha512-3ouUOpQhtgrbOa17J7+uxOTpITYWaGP7/AhoR3+A+/1e9skrzelGi/dXzEYyvbxubEF6Wn2ypscTKiKJFFn1ag==",
"dev": true
},
"node_modules/prompts": { "node_modules/prompts": {
"version": "2.4.2", "version": "2.4.2",
"resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.2.tgz", "resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.2.tgz",
@ -4651,18 +4578,6 @@
"node": ">=8" "node": ">=8"
} }
}, },
"node_modules/readable-stream": {
"version": "1.0.34",
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-1.0.34.tgz",
"integrity": "sha1-Elgg40vIQtLyqq+v5MKRbuMsFXw=",
"dev": true,
"dependencies": {
"core-util-is": "~1.0.0",
"inherits": "~2.0.1",
"isarray": "0.0.1",
"string_decoder": "~0.10.x"
}
},
"node_modules/redent": { "node_modules/redent": {
"version": "3.0.0", "version": "3.0.0",
"resolved": "https://registry.npmjs.org/redent/-/redent-3.0.0.tgz", "resolved": "https://registry.npmjs.org/redent/-/redent-3.0.0.tgz",
@ -4938,12 +4853,6 @@
"node": ">=8" "node": ">=8"
} }
}, },
"node_modules/string_decoder": {
"version": "0.10.31",
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-0.10.31.tgz",
"integrity": "sha1-YuIDvEF2bGwoyfyEMB2rHFMQ+pQ=",
"dev": true
},
"node_modules/string-argv": { "node_modules/string-argv": {
"version": "0.3.1", "version": "0.3.1",
"resolved": "https://registry.npmjs.org/string-argv/-/string-argv-0.3.1.tgz", "resolved": "https://registry.npmjs.org/string-argv/-/string-argv-0.3.1.tgz",
@ -5101,46 +5010,6 @@
"integrity": "sha512-8hmiGIJMDlwjg7dlJ4yKGLK8EsYqKgPWbG3b4wjJddKNwc7N7Dpn08Df4szr/sZdMVeOstrdYSsqzX6BYbcB+w==", "integrity": "sha512-8hmiGIJMDlwjg7dlJ4yKGLK8EsYqKgPWbG3b4wjJddKNwc7N7Dpn08Df4szr/sZdMVeOstrdYSsqzX6BYbcB+w==",
"dev": true "dev": true
}, },
"node_modules/through2": {
"version": "2.0.5",
"resolved": "https://registry.npmjs.org/through2/-/through2-2.0.5.tgz",
"integrity": "sha512-/mrRod8xqpA+IHSLyGCQ2s8SPHiCDEeQJSep1jqLYeEUClOFG2Qsh+4FU6G9VeqpZnGW/Su8LQGc4YKni5rYSQ==",
"dev": true,
"dependencies": {
"readable-stream": "~2.3.6",
"xtend": "~4.0.1"
}
},
"node_modules/through2/node_modules/isarray": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/isarray/-/isarray-1.0.0.tgz",
"integrity": "sha1-u5NdSFgsuhaMBoNJV6VKPgcSTxE=",
"dev": true
},
"node_modules/through2/node_modules/readable-stream": {
"version": "2.3.7",
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.7.tgz",
"integrity": "sha512-Ebho8K4jIbHAxnuxi7o42OrZgF/ZTNcsZj6nRKyUmkhLFq8CHItp/fy6hQZuZmP/n3yZ9VBUbp4zz/mX8hmYPw==",
"dev": true,
"dependencies": {
"core-util-is": "~1.0.0",
"inherits": "~2.0.3",
"isarray": "~1.0.0",
"process-nextick-args": "~2.0.0",
"safe-buffer": "~5.1.1",
"string_decoder": "~1.1.1",
"util-deprecate": "~1.0.1"
}
},
"node_modules/through2/node_modules/string_decoder": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.1.1.tgz",
"integrity": "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg==",
"dev": true,
"dependencies": {
"safe-buffer": "~5.1.0"
}
},
"node_modules/timsort": { "node_modules/timsort": {
"version": "0.3.0", "version": "0.3.0",
"resolved": "https://registry.npmjs.org/timsort/-/timsort-0.3.0.tgz", "resolved": "https://registry.npmjs.org/timsort/-/timsort-0.3.0.tgz",
@ -5375,15 +5244,6 @@
"node": ">= 4.0.0" "node": ">= 4.0.0"
} }
}, },
"node_modules/untildify": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/untildify/-/untildify-4.0.0.tgz",
"integrity": "sha512-KK8xQ1mkzZeg9inewmFVDNkg3l5LUhoq9kN6iWYB/CC9YMG8HA+c1Q8HwDe6dEX7kErrEVNVBO3fWsVq5iDgtw==",
"dev": true,
"engines": {
"node": ">=8"
}
},
"node_modules/update-browserslist-db": { "node_modules/update-browserslist-db": {
"version": "1.0.4", "version": "1.0.4",
"resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.0.4.tgz", "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.0.4.tgz",
@ -5419,12 +5279,6 @@
"punycode": "^2.1.0" "punycode": "^2.1.0"
} }
}, },
"node_modules/util-deprecate": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
"integrity": "sha1-RQ1Nyfpw3nMnYvvS1KKJgUGaDM8=",
"dev": true
},
"node_modules/v8-compile-cache": { "node_modules/v8-compile-cache": {
"version": "2.3.0", "version": "2.3.0",
"resolved": "https://registry.npmjs.org/v8-compile-cache/-/v8-compile-cache-2.3.0.tgz", "resolved": "https://registry.npmjs.org/v8-compile-cache/-/v8-compile-cache-2.3.0.tgz",
@ -5539,15 +5393,6 @@
"node": "^12.13.0 || ^14.15.0 || >=16" "node": "^12.13.0 || ^14.15.0 || >=16"
} }
}, },
"node_modules/xtend": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
"integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
"dev": true,
"engines": {
"node": ">=0.4"
}
},
"node_modules/y18n": { "node_modules/y18n": {
"version": "5.0.5", "version": "5.0.5",
"resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.5.tgz", "resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.5.tgz",
@ -5563,24 +5408,6 @@
"integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==", "integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==",
"dev": true "dev": true
}, },
"node_modules/yargs": {
"version": "16.2.0",
"resolved": "https://registry.npmjs.org/yargs/-/yargs-16.2.0.tgz",
"integrity": "sha512-D1mvvtDG0L5ft/jGWkLpG1+m0eQxOfaBvTNELraWj22wSVUMWxZUvYgJYcKh6jGGIkJFhH4IZPQhR4TKpc8mBw==",
"dev": true,
"dependencies": {
"cliui": "^7.0.2",
"escalade": "^3.1.1",
"get-caller-file": "^2.0.5",
"require-directory": "^2.1.1",
"string-width": "^4.2.0",
"y18n": "^5.0.5",
"yargs-parser": "^20.2.2"
},
"engines": {
"node": ">=10"
}
},
"node_modules/yargs-parser": { "node_modules/yargs-parser": {
"version": "20.2.4", "version": "20.2.4",
"resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-20.2.4.tgz", "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-20.2.4.tgz",
@ -7245,27 +7072,6 @@
"safe-buffer": "~5.1.1" "safe-buffer": "~5.1.1"
} }
}, },
"copyfiles": {
"version": "2.4.1",
"resolved": "https://registry.npmjs.org/copyfiles/-/copyfiles-2.4.1.tgz",
"integrity": "sha512-fereAvAvxDrQDOXybk3Qu3dPbOoKoysFMWtkY3mv5BsL8//OSZVL5DCLYqgRfY5cWirgRzlC+WSrxp6Bo3eNZg==",
"dev": true,
"requires": {
"glob": "^7.0.5",
"minimatch": "^3.0.3",
"mkdirp": "^1.0.4",
"noms": "0.0.0",
"through2": "^2.0.1",
"untildify": "^4.0.0",
"yargs": "^16.1.0"
}
},
"core-util-is": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.2.tgz",
"integrity": "sha1-tf1UIgqivFq1eqtxQMlAdUUDwac=",
"dev": true
},
"create-require": { "create-require": {
"version": "1.1.1", "version": "1.1.1",
"resolved": "https://registry.npmjs.org/create-require/-/create-require-1.1.1.tgz", "resolved": "https://registry.npmjs.org/create-require/-/create-require-1.1.1.tgz",
@ -8014,12 +7820,6 @@
"integrity": "sha512-knxG2q4UC3u8stRGyAVJCOdxFmv5DZiRcdlIaAQXAbSfJya+OhopNotLQrstBhququ4ZpuKbDc/8S6mgXgPFPw==", "integrity": "sha512-knxG2q4UC3u8stRGyAVJCOdxFmv5DZiRcdlIaAQXAbSfJya+OhopNotLQrstBhququ4ZpuKbDc/8S6mgXgPFPw==",
"dev": true "dev": true
}, },
"isarray": {
"version": "0.0.1",
"resolved": "https://registry.npmjs.org/isarray/-/isarray-0.0.1.tgz",
"integrity": "sha1-ihis/Kmo9Bd+Cav8YDiTmwXR7t8=",
"dev": true
},
"isexe": { "isexe": {
"version": "2.0.0", "version": "2.0.0",
"resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz", "resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz",
@ -8814,12 +8614,6 @@
"kind-of": "^6.0.3" "kind-of": "^6.0.3"
} }
}, },
"mkdirp": {
"version": "1.0.4",
"resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-1.0.4.tgz",
"integrity": "sha512-vVqVZQyf3WLx2Shd0qJ9xuvqgAyKPLAiqITEtqW0oIUjzo3PePDd6fW9iFz30ef7Ysp/oiWqbhszeGWW2T6Gzw==",
"dev": true
},
"ms": { "ms": {
"version": "2.1.2", "version": "2.1.2",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.2.tgz", "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.2.tgz",
@ -8844,16 +8638,6 @@
"integrity": "sha512-PiVXnNuFm5+iYkLBNeq5211hvO38y63T0i2KKh2KnUs3RpzJ+JtODFjkD8yjLwnDkTYF1eKXheUwdssR+NRZdg==", "integrity": "sha512-PiVXnNuFm5+iYkLBNeq5211hvO38y63T0i2KKh2KnUs3RpzJ+JtODFjkD8yjLwnDkTYF1eKXheUwdssR+NRZdg==",
"dev": true "dev": true
}, },
"noms": {
"version": "0.0.0",
"resolved": "https://registry.npmjs.org/noms/-/noms-0.0.0.tgz",
"integrity": "sha1-2o69nzr51nYJGbJ9nNyAkqczKFk=",
"dev": true,
"requires": {
"inherits": "^2.0.1",
"readable-stream": "~1.0.31"
}
},
"normalize-package-data": { "normalize-package-data": {
"version": "3.0.2", "version": "3.0.2",
"resolved": "https://registry.npmjs.org/normalize-package-data/-/normalize-package-data-3.0.2.tgz", "resolved": "https://registry.npmjs.org/normalize-package-data/-/normalize-package-data-3.0.2.tgz",
@ -8988,12 +8772,6 @@
"integrity": "sha512-gDKb8aZMDeD/tZWs9P6+q0J9Mwkdl6xMV8TjnGP3qJVJ06bdMgkbBlLU8IdfOsIsFz2BW1rNVT3XuNEl8zPAvw==", "integrity": "sha512-gDKb8aZMDeD/tZWs9P6+q0J9Mwkdl6xMV8TjnGP3qJVJ06bdMgkbBlLU8IdfOsIsFz2BW1rNVT3XuNEl8zPAvw==",
"dev": true "dev": true
}, },
"peggy": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/peggy/-/peggy-1.2.0.tgz",
"integrity": "sha512-PQ+NKpAobImfMprYQtc4Egmyi29bidRGEX0kKjCU5uuW09s0Cthwqhfy7mLkwcB4VcgacE5L/ZjruD/kOPCUUw==",
"dev": true
},
"picocolors": { "picocolors": {
"version": "1.0.0", "version": "1.0.0",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.0.0.tgz", "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.0.0.tgz",
@ -9056,12 +8834,6 @@
} }
} }
}, },
"process-nextick-args": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-2.0.1.tgz",
"integrity": "sha512-3ouUOpQhtgrbOa17J7+uxOTpITYWaGP7/AhoR3+A+/1e9skrzelGi/dXzEYyvbxubEF6Wn2ypscTKiKJFFn1ag==",
"dev": true
},
"prompts": { "prompts": {
"version": "2.4.2", "version": "2.4.2",
"resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.2.tgz", "resolved": "https://registry.npmjs.org/prompts/-/prompts-2.4.2.tgz",
@ -9159,18 +8931,6 @@
} }
} }
}, },
"readable-stream": {
"version": "1.0.34",
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-1.0.34.tgz",
"integrity": "sha1-Elgg40vIQtLyqq+v5MKRbuMsFXw=",
"dev": true,
"requires": {
"core-util-is": "~1.0.0",
"inherits": "~2.0.1",
"isarray": "0.0.1",
"string_decoder": "~0.10.x"
}
},
"redent": { "redent": {
"version": "3.0.0", "version": "3.0.0",
"resolved": "https://registry.npmjs.org/redent/-/redent-3.0.0.tgz", "resolved": "https://registry.npmjs.org/redent/-/redent-3.0.0.tgz",
@ -9375,12 +9135,6 @@
} }
} }
}, },
"string_decoder": {
"version": "0.10.31",
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-0.10.31.tgz",
"integrity": "sha1-YuIDvEF2bGwoyfyEMB2rHFMQ+pQ=",
"dev": true
},
"string-argv": { "string-argv": {
"version": "0.3.1", "version": "0.3.1",
"resolved": "https://registry.npmjs.org/string-argv/-/string-argv-0.3.1.tgz", "resolved": "https://registry.npmjs.org/string-argv/-/string-argv-0.3.1.tgz",
@ -9496,48 +9250,6 @@
"integrity": "sha512-8hmiGIJMDlwjg7dlJ4yKGLK8EsYqKgPWbG3b4wjJddKNwc7N7Dpn08Df4szr/sZdMVeOstrdYSsqzX6BYbcB+w==", "integrity": "sha512-8hmiGIJMDlwjg7dlJ4yKGLK8EsYqKgPWbG3b4wjJddKNwc7N7Dpn08Df4szr/sZdMVeOstrdYSsqzX6BYbcB+w==",
"dev": true "dev": true
}, },
"through2": {
"version": "2.0.5",
"resolved": "https://registry.npmjs.org/through2/-/through2-2.0.5.tgz",
"integrity": "sha512-/mrRod8xqpA+IHSLyGCQ2s8SPHiCDEeQJSep1jqLYeEUClOFG2Qsh+4FU6G9VeqpZnGW/Su8LQGc4YKni5rYSQ==",
"dev": true,
"requires": {
"readable-stream": "~2.3.6",
"xtend": "~4.0.1"
},
"dependencies": {
"isarray": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/isarray/-/isarray-1.0.0.tgz",
"integrity": "sha1-u5NdSFgsuhaMBoNJV6VKPgcSTxE=",
"dev": true
},
"readable-stream": {
"version": "2.3.7",
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.7.tgz",
"integrity": "sha512-Ebho8K4jIbHAxnuxi7o42OrZgF/ZTNcsZj6nRKyUmkhLFq8CHItp/fy6hQZuZmP/n3yZ9VBUbp4zz/mX8hmYPw==",
"dev": true,
"requires": {
"core-util-is": "~1.0.0",
"inherits": "~2.0.3",
"isarray": "~1.0.0",
"process-nextick-args": "~2.0.0",
"safe-buffer": "~5.1.1",
"string_decoder": "~1.1.1",
"util-deprecate": "~1.0.1"
}
},
"string_decoder": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.1.1.tgz",
"integrity": "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg==",
"dev": true,
"requires": {
"safe-buffer": "~5.1.0"
}
}
}
},
"timsort": { "timsort": {
"version": "0.3.0", "version": "0.3.0",
"resolved": "https://registry.npmjs.org/timsort/-/timsort-0.3.0.tgz", "resolved": "https://registry.npmjs.org/timsort/-/timsort-0.3.0.tgz",
@ -9683,12 +9395,6 @@
"integrity": "sha512-rBJeI5CXAlmy1pV+617WB9J63U6XcazHHF2f2dbJix4XzpUF0RS3Zbj0FGIOCAva5P/d/GBOYaACQ1w+0azUkg==", "integrity": "sha512-rBJeI5CXAlmy1pV+617WB9J63U6XcazHHF2f2dbJix4XzpUF0RS3Zbj0FGIOCAva5P/d/GBOYaACQ1w+0azUkg==",
"dev": true "dev": true
}, },
"untildify": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/untildify/-/untildify-4.0.0.tgz",
"integrity": "sha512-KK8xQ1mkzZeg9inewmFVDNkg3l5LUhoq9kN6iWYB/CC9YMG8HA+c1Q8HwDe6dEX7kErrEVNVBO3fWsVq5iDgtw==",
"dev": true
},
"update-browserslist-db": { "update-browserslist-db": {
"version": "1.0.4", "version": "1.0.4",
"resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.0.4.tgz", "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.0.4.tgz",
@ -9708,12 +9414,6 @@
"punycode": "^2.1.0" "punycode": "^2.1.0"
} }
}, },
"util-deprecate": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
"integrity": "sha1-RQ1Nyfpw3nMnYvvS1KKJgUGaDM8=",
"dev": true
},
"v8-compile-cache": { "v8-compile-cache": {
"version": "2.3.0", "version": "2.3.0",
"resolved": "https://registry.npmjs.org/v8-compile-cache/-/v8-compile-cache-2.3.0.tgz", "resolved": "https://registry.npmjs.org/v8-compile-cache/-/v8-compile-cache-2.3.0.tgz",
@ -9804,12 +9504,6 @@
"signal-exit": "^3.0.7" "signal-exit": "^3.0.7"
} }
}, },
"xtend": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
"integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
"dev": true
},
"y18n": { "y18n": {
"version": "5.0.5", "version": "5.0.5",
"resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.5.tgz", "resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.5.tgz",
@ -9822,21 +9516,6 @@
"integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==", "integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==",
"dev": true "dev": true
}, },
"yargs": {
"version": "16.2.0",
"resolved": "https://registry.npmjs.org/yargs/-/yargs-16.2.0.tgz",
"integrity": "sha512-D1mvvtDG0L5ft/jGWkLpG1+m0eQxOfaBvTNELraWj22wSVUMWxZUvYgJYcKh6jGGIkJFhH4IZPQhR4TKpc8mBw==",
"dev": true,
"requires": {
"cliui": "^7.0.2",
"escalade": "^3.1.1",
"get-caller-file": "^2.0.5",
"require-directory": "^2.1.1",
"string-width": "^4.2.0",
"y18n": "^5.0.5",
"yargs-parser": "^20.2.2"
}
},
"yargs-parser": { "yargs-parser": {
"version": "20.2.4", "version": "20.2.4",
"resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-20.2.4.tgz", "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-20.2.4.tgz",

12
package.json
View file

@ -1,15 +1,11 @@
{ {
"name": "mfm-js", "name": "mfm-js",
"version": "0.22.1", "version": "0.23.0-canary.1",
"description": "An MFM parser implementation with PEG.js", "description": "An MFM parser implementation with TypeScript",
"main": "./built/index.js", "main": "./built/index.js",
"types": "./built/index.d.ts", "types": "./built/index.d.ts",
"scripts": { "scripts": {
"build": "npm run tsc && npm run peg", "build": "npm run tsc",
"build-debug": "npm run tsc && npm run peg-debug",
"peg": "peggy --cache -o src/internal/parser.js --allowed-start-rules fullParser,simpleParser src/internal/parser.pegjs && npm run peg-copy",
"peg-debug": "peggy --cache -o src/internal/parser.js --allowed-start-rules fullParser,inlineParser,simpleParser --trace src/internal/parser.pegjs && npm run peg-copy",
"peg-copy": "copyfiles -f src/internal/parser.js built/internal/",
"tsc": "tsc", "tsc": "tsc",
"tsd": "tsd", "tsd": "tsd",
"parse": "node ./built/cli/parse", "parse": "node ./built/cli/parse",
@ -32,10 +28,8 @@
"@types/node": "18.0.3", "@types/node": "18.0.3",
"@typescript-eslint/eslint-plugin": "^5.30.5", "@typescript-eslint/eslint-plugin": "^5.30.5",
"@typescript-eslint/parser": "^5.30.5", "@typescript-eslint/parser": "^5.30.5",
"copyfiles": "^2.4.1",
"eslint": "^8.19.0", "eslint": "^8.19.0",
"jest": "^28.1.2", "jest": "^28.1.2",
"peggy": "1.2.0",
"ts-jest": "^28.0.5", "ts-jest": "^28.0.5",
"ts-node": "10.8.2", "ts-node": "10.8.2",
"tsd": "^0.22.0", "tsd": "^0.22.0",

4
src/@types/twemoji.d.ts vendored Normal file
View file

@ -0,0 +1,4 @@
declare module 'twemoji-parser/dist/lib/regex' {
const regex: RegExp;
export default regex;
}

12
src/api.ts
View file

@ -1,16 +1,12 @@
import peg from 'peggy'; import { fullParser, simpleParser } from './internal';
import { inspectOne, stringifyNode, stringifyTree } from './internal/util';
import { MfmNode, MfmSimpleNode } from './node'; import { MfmNode, MfmSimpleNode } from './node';
import { stringifyNode, stringifyTree, inspectOne } from './internal/util';
// eslint-disable-next-line @typescript-eslint/no-var-requires
const parser: peg.Parser = require('./internal/parser');
/** /**
* Generates a MfmNode tree from the MFM string. * Generates a MfmNode tree from the MFM string.
*/ */
export function parse(input: string, opts: Partial<{ fnNameList: string[]; nestLimit: number; }> = {}): MfmNode[] { export function parse(input: string, opts: Partial<{ fnNameList: string[]; nestLimit: number; }> = {}): MfmNode[] {
const nodes = parser.parse(input, { const nodes = fullParser(input, {
startRule: 'fullParser',
fnNameList: opts.fnNameList, fnNameList: opts.fnNameList,
nestLimit: opts.nestLimit, nestLimit: opts.nestLimit,
}); });
@ -21,7 +17,7 @@ export function parse(input: string, opts: Partial<{ fnNameList: string[]; nestL
* Generates a MfmSimpleNode tree from the MFM string. * Generates a MfmSimpleNode tree from the MFM string.
*/ */
export function parseSimple(input: string): MfmSimpleNode[] { export function parseSimple(input: string): MfmSimpleNode[] {
const nodes = parser.parse(input, { startRule: 'simpleParser' }); const nodes = simpleParser(input);
return nodes; return nodes;
} }

249
src/internal/core/index.ts Normal file
View file

@ -0,0 +1,249 @@
//
// Parsimmon-like stateful parser combinators
//
export type Success<T> = {
success: true;
value: T;
index: number;
};
export type Failure = { success: false };
export type Result<T> = Success<T> | Failure;
export type ParserHandler<T> = (input: string, index: number, state: any) => Result<T>
export function success<T>(index: number, value: T): Success<T> {
return {
success: true,
value: value,
index: index,
};
}
export function failure(): Failure {
return { success: false };
}
export class Parser<T> {
public name?: string;
public handler: ParserHandler<T>;
constructor(handler: ParserHandler<T>, name?: string) {
this.handler = (input, index, state) => {
if (state.trace && this.name != null) {
const pos = `${index}`;
console.log(`${pos.padEnd(6, ' ')}enter ${this.name}`);
const result = handler(input, index, state);
if (result.success) {
const pos = `${index}:${result.index}`;
console.log(`${pos.padEnd(6, ' ')}match ${this.name}`);
} else {
const pos = `${index}`;
console.log(`${pos.padEnd(6, ' ')}fail ${this.name}`);
}
return result;
}
return handler(input, index, state);
};
this.name = name;
}
map<U>(fn: (value: T) => U): Parser<U> {
return new Parser((input, index, state) => {
const result = this.handler(input, index, state);
if (!result.success) {
return result;
}
return success(result.index, fn(result.value));
});
}
text(): Parser<string> {
return new Parser((input, index, state) => {
const result = this.handler(input, index, state);
if (!result.success) {
return result;
}
const text = input.slice(index, result.index);
return success(result.index, text);
});
}
many(min: number): Parser<T[]> {
return new Parser((input, index, state) => {
let result;
let latestIndex = index;
const accum: T[] = [];
while (latestIndex < input.length) {
result = this.handler(input, latestIndex, state);
if (!result.success) {
break;
}
latestIndex = result.index;
accum.push(result.value);
}
if (accum.length < min) {
return failure();
}
return success(latestIndex, accum);
});
}
sep(separator: Parser<any>, min: number): Parser<T[]> {
if (min < 1) {
throw new Error('"min" must be a value greater than or equal to 1.');
}
return seq([
this,
seq([
separator,
this,
], 1).many(min - 1),
]).map(result => [result[0], ...result[1]]);
}
option<T>(): Parser<T | null> {
return alt([
this,
succeeded(null),
]);
}
}
export function str<T extends string>(value: T): Parser<T> {
return new Parser((input, index, _state) => {
if ((input.length - index) < value.length) {
return failure();
}
if (input.substr(index, value.length) !== value) {
return failure();
}
return success(index + value.length, value);
});
}
export function regexp<T extends RegExp>(pattern: T): Parser<string> {
const re = RegExp(`^(?:${pattern.source})`, pattern.flags);
return new Parser((input, index, _state) => {
const text = input.slice(index);
const result = re.exec(text);
if (result == null) {
return failure();
}
return success(index + result[0].length, result[0]);
});
}
export function seq(parsers: Parser<any>[], select?: number): Parser<any> {
return new Parser((input, index, state) => {
let result;
let latestIndex = index;
const accum = [];
for (let i = 0; i < parsers.length; i++) {
result = parsers[i].handler(input, latestIndex, state);
if (!result.success) {
return result;
}
latestIndex = result.index;
accum.push(result.value);
}
return success(latestIndex, (select != null ? accum[select] : accum));
});
}
export function alt(parsers: Parser<any>[]): Parser<any> {
return new Parser((input, index, state) => {
let result;
for (let i = 0; i < parsers.length; i++) {
result = parsers[i].handler(input, index, state);
if (result.success) {
return result;
}
}
return failure();
});
}
function succeeded<T>(value: T): Parser<T> {
return new Parser((_input, index, _state) => {
return success(index, value);
});
}
export function notMatch(parser: Parser<any>): Parser<null> {
return new Parser((input, index, state) => {
const result = parser.handler(input, index, state);
return !result.success
? success(index, null)
: failure();
});
}
export const cr = str('\r');
export const lf = str('\n');
export const crlf = str('\r\n');
export const newline = alt([crlf, cr, lf]);
export const char = new Parser((input, index, _state) => {
if ((input.length - index) < 1) {
return failure();
}
const value = input.charAt(index);
return success(index + 1, value);
});
export const lineBegin = new Parser((input, index, state) => {
if (index === 0) {
return success(index, null);
}
if (cr.handler(input, index - 1, state).success) {
return success(index, null);
}
if (lf.handler(input, index - 1, state).success) {
return success(index, null);
}
return failure();
});
export const lineEnd = new Parser((input, index, state) => {
if (index === input.length) {
return success(index, null);
}
if (cr.handler(input, index, state).success) {
return success(index, null);
}
if (lf.handler(input, index, state).success) {
return success(index, null);
}
return failure();
});
export function lazy<T>(fn: () => Parser<T>): Parser<T> {
const parser: Parser<T> = new Parser((input, index, state) => {
parser.handler = fn().handler;
return parser.handler(input, index, state);
});
return parser;
}
//type Syntax<T> = (rules: Record<string, Parser<T>>) => Parser<T>;
//type SyntaxReturn<T> = T extends (rules: Record<string, Parser<any>>) => infer R ? R : never;
//export function createLanguage2<T extends Record<string, Syntax<any>>>(syntaxes: T): { [K in keyof T]: SyntaxReturn<T[K]> } {
// TODO: 関数の型宣言をいい感じにしたい
export function createLanguage<T>(syntaxes: { [K in keyof T]: (r: Record<string, Parser<any>>) => T[K] }): T {
const rules: Record<string, Parser<any>> = {};
for (const key of Object.keys(syntaxes)) {
rules[key] = lazy(() => {
const parser = (syntaxes as any)[key](rules);
if (parser == null) {
throw new Error('syntax must return a parser.');
}
parser.name = key;
return parser;
});
}
return rules as any;
}

25
src/internal/index.ts Normal file
View file

@ -0,0 +1,25 @@
import * as M from '..';
import { language } from './parser';
import { mergeText } from './util';
import * as P from './core';
export type FullParserOpts = {
fnNameList?: string[];
nestLimit?: number;
};
export function fullParser(input: string, opts: FullParserOpts): M.MfmNode[] {
const result = language.fullParser.handler(input, 0, {
nestLimit: (opts.nestLimit != null) ? opts.nestLimit : 20,
fnNameList: opts.fnNameList,
depth: 0,
linkLabel: false,
trace: false,
}) as P.Success<any>;
return mergeText(result.value);
}
export function simpleParser(input: string): M.MfmSimpleNode[] {
const result = language.simpleParser.handler(input, 0, { }) as P.Success<any>;
return mergeText(result.value);
}

609
src/internal/parser.pegjs
View file

@ -1,609 +0,0 @@
{
const {
// block
QUOTE,
SEARCH,
CODE_BLOCK,
MATH_BLOCK,
CENTER,
// inline
UNI_EMOJI,
EMOJI_CODE,
BOLD,
SMALL,
ITALIC,
STRIKE,
INLINE_CODE,
MATH_INLINE,
MENTION,
HASHTAG,
N_URL,
LINK,
FN,
PLAIN,
TEXT
} = require('../node');
const {
mergeText,
setConsumeCount,
consumeDynamically
} = require('./util');
function applyParser(input, startRule, opts) {
const parseFunc = peg$parse;
const parseOpts = {
fnNameList: options.fnNameList,
nestLimit: (nestLimit - depth),
...(opts || {}),
};
if (startRule) parseOpts.startRule = startRule;
return parseFunc(input, parseOpts);
}
// emoji
const emojiRegex = require('twemoji-parser/dist/lib/regex').default;
const anchoredEmojiRegex = RegExp(`^(?:${emojiRegex.source})`);
/**
* check if the input matches the emoji regexp.
* if they match, set the byte length of the emoji.
*/
function matchUnicodeEmoji() {
const offset = location().start.offset;
const src = input.substr(offset);
const result = anchoredEmojiRegex.exec(src);
if (result != null) {
setConsumeCount(result[0].length); // length(utf-16 byte length) of emoji sequence.
return true;
}
return false;
}
function ensureFnName(name) {
if (!options.fnNameList) return true;
if (!Array.isArray(options.fnNameList)) {
error("options.fnNameList must be an array.");
}
return options.fnNameList.includes(name);
}
// nesting control
const nestLimit = (options.nestLimit != null ? options.nestLimit : 20);
let depth = 0;
function enterNest() {
if (depth + 1 > nestLimit) {
return false;
}
depth++;
return true;
}
function leaveNest() {
depth--;
return true;
}
function fallbackNest() {
depth--;
return false;
}
}
//
// parsers
//
fullParser
= nodes:(&. @full)* { return mergeText(nodes); }
simpleParser
= nodes:(&. @simple)* { return mergeText(nodes); }
//
// syntax list
//
full
= quote // block
/ codeBlock // block
/ mathBlock // block
/ center // block
/ emojiCode
/ unicodeEmoji
/ big
/ bold
/ small
/ italic
/ strike
/ inlineCode
/ mathInline
/ mention
/ hashtag
/ url
/ fn
/ plain
/ link
/ search // block
/ inlineText
inline
= emojiCode
/ unicodeEmoji
/ big
/ bold
/ small
/ italic
/ strike
/ inlineCode
/ mathInline
/ mention
/ hashtag
/ url
/ fn
/ plain
/ link
/ inlineText
L_inline
= emojiCode
/ unicodeEmoji
/ L_big
/ L_bold
/ L_small
/ L_italic
/ L_strike
/ inlineCode
/ mathInline
/ L_fn
/ plain
/ L_inlineText
simple
= emojiCode
/ unicodeEmoji
/ simpleText
//
// block rules
//
// block: quote
quote
= &(BEGIN ">") &{ return (depth + 1 <= nestLimit); } @quoteInner LF?
quoteInner
= head:(quoteLine / quoteEmptyLine) tails:(quoteLine / quoteEmptyLine)+
{
depth++;
const children = applyParser([head, ...tails].join('\n'), 'fullParser');
depth--;
return QUOTE(children);
}
/ line:quoteLine
{
depth++;
const children = applyParser(line, 'fullParser');
depth--;
return QUOTE(children);
}
quoteLine
= BEGIN ">" _? @$CHAR+ END
quoteEmptyLine
= BEGIN ">" _? END
{
return '';
}
// block: search
search
= BEGIN q:searchQuery sp:_ key:searchKey END
{
return SEARCH(q, `${ q }${ sp }${ key }`);
}
searchQuery
= (!(_ searchKey END) CHAR)+ { return text(); }
searchKey
= "[" ("検索" / "Search"i) "]" { return text(); }
/ "検索"
/ "Search"i
// block: codeBlock
codeBlock
= BEGIN "```" lang:$(CHAR*) LF code:codeBlockContent LF "```" END
{
lang = lang.trim();
return CODE_BLOCK(code, lang.length > 0 ? lang : null);
}
codeBlockContent
= (!(LF "```" END) .)+
{ return text(); }
// block: mathBlock
mathBlock
= BEGIN "\\[" LF? formula:mathBlockLines LF? "\\]" END
{
return MATH_BLOCK(formula.trim());
}
mathBlockLines
= mathBlockLine (LF mathBlockLine)*
{ return text(); }
mathBlockLine
= (!"\\]" CHAR)+
// block: center
center
= BEGIN "<center>" LF? content:(!(LF? "</center>" END) @inline)* LF? "</center>" END
{
return CENTER(mergeText(content));
}
//
// inline rules
//
// inline: emoji code
emojiCode
= ":" name:$[a-z0-9_+-]i+ ":"
{
return EMOJI_CODE(name);
}
// inline: unicode emoji
// NOTE: if the text matches one of the emojis, it will count the length of the emoji sequence and consume it.
unicodeEmoji
= &{ return matchUnicodeEmoji(); } (&{ return consumeDynamically(); } .)+
{
return UNI_EMOJI(text());
}
// inline: big
big
= "***" content:bigContent "***"
{
return FN('tada', { }, mergeText(content));
}
bigContent
= &{ return enterNest(); } @(@(!"***" @inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
L_big
= "***" content:L_bigContent "***"
{
return FN('tada', { }, mergeText(content));
}
L_bigContent
= &{ return enterNest(); } @(@(!"***" @L_inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
// inline: bold
bold
= "**" content:boldContent "**"
{
return BOLD(mergeText(content));
}
/ "<b>" content:boldTagContent "</b>"
{
return BOLD(mergeText(content));
}
/ "__" content:$(!"__" @([a-z0-9]i / _))+ "__"
{
return BOLD([TEXT(content)]);
}
boldContent
= &{ return enterNest(); } @(@(!"**" @inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
boldTagContent
= &{ return enterNest(); } @(@(!"</b>" @inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
L_bold
= "**" content:L_boldContent "**"
{
return BOLD(mergeText(content));
}
/ "<b>" content:L_boldTagContent "</b>"
{
return BOLD(mergeText(content));
}
/ "__" content:$(!"__" @([a-z0-9]i / _))+ "__"
{
return BOLD([TEXT(content)]);
}
L_boldContent
= &{ return enterNest(); } @(@(!"**" @L_inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
L_boldTagContent
= &{ return enterNest(); } @(@(!"</b>" @L_inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
// inline: small
small
= "<small>" content:smallContent "</small>"
{
return SMALL(mergeText(content));
}
smallContent
= &{ return enterNest(); } @(@(!"</small>" @inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
L_small
= "<small>" content:L_smallContent "</small>"
{
return SMALL(mergeText(content));
}
L_smallContent
= &{ return enterNest(); } @(@(!"</small>" @L_inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
// inline: italic
italic
= "<i>" content:italicContent "</i>"
{
return ITALIC(mergeText(content));
}
/ italicAlt
L_italic
= "<i>" content:L_italicContent "</i>"
{
return ITALIC(mergeText(content));
}
/ italicAlt
italicAlt
= "*" content:$([a-z0-9]i / _)+ "*" &(EOF / LF / _ / ![a-z0-9]i)
{
return ITALIC([TEXT(content)]);
}
/ "_" content:$([a-z0-9]i / _)+ "_" &(EOF / LF / _ / ![a-z0-9]i)
{
return ITALIC([TEXT(content)]);
}
italicContent
= &{ return enterNest(); } @(@(!"</i>" @inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
L_italicContent
= &{ return enterNest(); } @(@(!"</i>" @L_inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
// inline: strike
strike
= "~~" content:strikeContent "~~"
{
return STRIKE(mergeText(content));
}
/ "<s>" content:strikeTagContent "</s>"
{
return STRIKE(mergeText(content));
}
strikeContent
= &{ return enterNest(); } @(@(!("~" / LF) @inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
strikeTagContent
= &{ return enterNest(); } @(@(!"</s>" @inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
L_strike
= "~~" content:L_strikeContent "~~"
{
return STRIKE(mergeText(content));
}
/ "<s>" content:L_strikeTagContent "</s>"
{
return STRIKE(mergeText(content));
}
L_strikeContent
= &{ return enterNest(); } @(@(!("~" / LF) @L_inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
L_strikeTagContent
= &{ return enterNest(); } @(@(!"</s>" @L_inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
// inline: inlineCode
inlineCode
= "`" content:$(![`´] CHAR)+ "`"
{
return INLINE_CODE(content);
}
// inline: mathInline
mathInline
= "\\(" content:$(!"\\)" CHAR)+ "\\)"
{
return MATH_INLINE(content);
}
// inline: mention
mention
= "@" name:mentionName host:("@" @mentionHost)?
{
return MENTION(name, host, text());
}
mentionName
= [a-z0-9_]i (&("-"+ [a-z0-9_]i) . / [a-z0-9_]i)*
{
// NOTE: first char and last char are not "-".
return text();
}
mentionHost
= [a-z0-9_]i (&([.-]i+ [a-z0-9_]i) . / [a-z0-9_]i)*
{
// NOTE: first char and last char are neither "." nor "-".
return text();
}
// inline: hashtag
hashtag
= "#" !("\uFE0F"? "\u20E3") !(invalidHashtagContent !hashtagContentPart) content:$hashtagContentPart+
{
return HASHTAG(content);
}
invalidHashtagContent
= [0-9]+
hashtagContentPart
= "(" hashPairInner ")"
/ "[" hashPairInner "]"
/ "「" hashPairInner "」"
/ "（" hashPairInner "）"
/ ![ 　\t.,!?'"#:\/\[\]【】()「」（）<>] CHAR
hashPairInner
= &{ return enterNest(); } @(@hashtagContentPart* &{ return leaveNest(); } / &{ return fallbackNest(); })
// inline: URL
url
= "<" url:$("http" "s"? "://" (!(">" / _) CHAR)+) ">"
{
return N_URL(url, true);
}
/ "http" "s"? "://" (&([.,]+ urlContentPart) . / urlContentPart)+
{
// NOTE: last char is neither "." nor ",".
return N_URL(text());
}
urlContentPart
= "(" urlPairInner ")"
/ "[" urlPairInner "]"
/ [a-z0-9_/:%#@$&?!~=+-]i
urlPairInner
= &{ return enterNest(); } @(@(urlContentPart / [.,])* &{ return leaveNest(); } / &{ return fallbackNest(); })
// inline: link
link
= silent:"?"? "[" label:linkLabel "](" url:url ")"
{
return LINK((silent != null), url.props.url, mergeText(label));
}
linkLabel
= (!"]" @L_inline)+
// inline: fn
fn
= "$[" name:$([a-z0-9_]i)+ &{ return ensureFnName(name); } args:fnArgs? _ content:fnContent "]"
{
args = args || {};
return FN(name, args, mergeText(content));
}
L_fn
= "$[" name:$([a-z0-9_]i)+ &{ return ensureFnName(name); } args:fnArgs? _ content:L_fnContent "]"
{
args = args || {};
return FN(name, args, mergeText(content));
}
fnContent
= &{ return enterNest(); } @(@(!"]" @inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
L_fnContent
= &{ return enterNest(); } @(@(!"]" @L_inline)+ &{ return leaveNest(); } / &{ return fallbackNest(); })
fnArgs
= "." head:fnArg tails:("," @fnArg)*
{
const args = { };
for (const pair of [head, ...tails]) {
args[pair.k] = pair.v;
}
return args;
}
fnArg
= k:$([a-z0-9_]i)+ "=" v:$([a-z0-9_.]i)+
{
return { k, v };
}
/ k:$([a-z0-9_]i)+
{
return { k: k, v: true };
}
// inline: plain
plain
= "<plain>" LF? content:plainContent LF? "</plain>"
{
return PLAIN(content);
}
plainContent
= (!(LF? "</plain>") .)+
{
return text();
}
// inline: text
inlineText
= !(LF / _) [a-z0-9]i &(hashtag / mention / italicAlt) . { return text(); } // hashtag, mention, italic ignore
/ . /* text node */
L_inlineText
= !(LF / _) [a-z0-9]i &italicAlt . { return text(); } // italic ignore
/ . /* text node */
// inline: text (for simpleParser)
simpleText
= . /* text node */
//
// General
//
BEGIN "beginning of line"
= LF / &{ return location().start.column == 1; }
END "end of line"
= LF / EOF
EOF
= !.
CHAR
= !LF . { return text(); }
LF
= "\r\n" / [\r\n]
_ "whitespace"
= [ 　\t\u00a0]

749
src/internal/parser.ts Normal file
View file

@ -0,0 +1,749 @@
import * as M from '..';
import * as P from './core';
import { mergeText } from './util';
// NOTE:
// tsdのテストでファイルを追加しているにも関わらず「twemoji-parser/dist/lib/regex」の型定義ファイルがないとエラーが出るため、
// このエラーを無視する。
/* eslint @typescript-eslint/ban-ts-comment: 1 */
// @ts-ignore
import twemojiRegex from 'twemoji-parser/dist/lib/regex';
type ArgPair = { k: string, v: string | true };
type Args = Record<string, string | true>;
const space = P.regexp(/[\u0020\u3000\t]/);
const alphaAndNum = P.regexp(/[a-z0-9]/i);
const newLine = P.alt([P.crlf, P.cr, P.lf]);
function seqOrText(parsers: P.Parser<any>[]): P.Parser<any[] | string> {
return new P.Parser<any[] | string>((input, index, state) => {
const accum: any[] = [];
let latestIndex = index;
for (let i = 0 ; i < parsers.length; i++) {
const result = parsers[i].handler(input, latestIndex, state);
if (!result.success) {
if (latestIndex === index) {
return P.failure();
} else {
return P.success(latestIndex, input.slice(index, latestIndex));
}
}
accum.push(result.value);
latestIndex = result.index;
}
return P.success(latestIndex, accum);
});
}
const notLinkLabel = new P.Parser((_input, index, state) => {
return (!state.linkLabel)
? P.success(index, null)
: P.failure();
});
const nestable = new P.Parser((_input, index, state) => {
return (state.depth < state.nestLimit)
? P.success(index, null)
: P.failure();
});
function nest<T>(parser: P.Parser<T>, fallback?: P.Parser<string>): P.Parser<T | string> {
// nesting limited? -> No: specified parser, Yes: fallback parser (default = P.char)
const inner = P.alt([
P.seq([nestable, parser], 1),
(fallback != null) ? fallback : P.char,
]);
return new P.Parser<T | string>((input, index, state) => {
state.depth++;
const result = inner.handler(input, index, state);
state.depth--;
return result;
});
}
export const language = P.createLanguage({
fullParser: r => {
return r.full.many(0);
},
simpleParser: r => {
return r.simple.many(0);
},
full: r => {
return P.alt([
// Regexp
r.unicodeEmoji,
// "<center>" block
r.centerTag,
// "<small>"
r.smallTag,
// "<plain>"
r.plainTag,
// "<b>"
r.boldTag,
// "<i>"
r.italicTag,
// "<s>"
r.strikeTag,
// "<http"
r.urlAlt,
// "***"
r.big,
// "**"
r.boldAsta,
// "*"
r.italicAsta,
// "__"
r.boldUnder,
// "_"
r.italicUnder,
// "```" block
r.codeBlock,
// "`"
r.inlineCode,
// ">" block
r.quote,
// "\\[" block
r.mathBlock,
// "\\("
r.mathInline,
// "~~"
r.strikeWave,
// "$[""
r.fn,
// "@"
r.mention,
// "#"
r.hashtag,
// ":"
r.emojiCode,
// "?[" or "["
r.link,
// http
r.url,
// block
r.search,
r.text,
]);
},
simple: r => {
return P.alt([
r.unicodeEmoji, // Regexp
r.emojiCode, // ":"
r.text,
]);
},
inline: r => {
return P.alt([
// Regexp
r.unicodeEmoji,
// "<small>"
r.smallTag,
// "<plain>"
r.plainTag,
// "<b>"
r.boldTag,
// "<i>"
r.italicTag,
// "<s>"
r.strikeTag,
// <http
r.urlAlt,
// "***"
r.big,
// "**"
r.boldAsta,
// "*"
r.italicAsta,
// "__"
r.boldUnder,
// "_"
r.italicUnder,
// "`"
r.inlineCode,
// "\\("
r.mathInline,
// "~~"
r.strikeWave,
// "$[""
r.fn,
// "@"
r.mention,
// "#"
r.hashtag,
// ":"
r.emojiCode,
// "?[" or "["
r.link,
// http
r.url,
r.text,
]);
},
quote: r => {
const lines: P.Parser<string[]> = P.seq([
P.str('>'),
space.option(),
P.seq([P.notMatch(newLine), P.char], 1).many(0).text(),
], 2).sep(newLine, 1);
const parser = P.seq([
newLine.option(),
newLine.option(),
P.lineBegin,
lines,
newLine.option(),
newLine.option(),
], 3);
return new P.Parser((input, index, state) => {
let result;
// parse quote
result = parser.handler(input, index, state);
if (!result.success) {
return result;
}
const contents = result.value;
const quoteIndex = result.index;
// disallow empty content if single line
if (contents.length === 1 && contents[0].length === 0) {
return P.failure();
}
// parse inner content
const contentParser = nest(r.fullParser).many(0);
result = contentParser.handler(contents.join('\n'), 0, state);
if (!result.success) {
return result;
}
return P.success(quoteIndex, M.QUOTE(mergeText(result.value)));
});
},
codeBlock: r => {
const mark = P.str('```');
return P.seq([
newLine.option(),
P.lineBegin,
mark,
P.seq([P.notMatch(newLine), P.char], 1).many(0),
newLine,
P.seq([P.notMatch(P.seq([newLine, mark, P.lineEnd])), P.char], 1).many(1),
newLine,
mark,
P.lineEnd,
newLine.option(),
]).map(result => {
const lang = (result[3] as string[]).join('').trim();
const code = (result[5] as string[]).join('');
return M.CODE_BLOCK(code, (lang.length > 0 ? lang : null));
});
},
mathBlock: r => {
const open = P.str('\\[');
const close = P.str('\\]');
return P.seq([
newLine.option(),
P.lineBegin,
open,
newLine.option(),
P.seq([P.notMatch(P.seq([newLine.option(), close])), P.char], 1).many(1),
newLine.option(),
close,
P.lineEnd,
newLine.option(),
]).map(result => {
const formula = (result[4] as string[]).join('');
return M.MATH_BLOCK(formula);
});
},
centerTag: r => {
const open = P.str('<center>');
const close = P.str('</center>');
return P.seq([
newLine.option(),
P.lineBegin,
open,
newLine.option(),
P.seq([P.notMatch(P.seq([newLine.option(), close])), nest(r.inline)], 1).many(1),
newLine.option(),
close,
P.lineEnd,
newLine.option(),
]).map(result => {
return M.CENTER(mergeText(result[4]));
});
},
big: r => {
const mark = P.str('***');
return seqOrText([
mark,
P.seq([P.notMatch(mark), nest(r.inline)], 1).many(1),
mark,
]).map(result => {
if (typeof result === 'string') return result;
return M.FN('tada', {}, mergeText(result[1]));
});
},
boldAsta: r => {
const mark = P.str('**');
return seqOrText([
mark,
P.seq([P.notMatch(mark), nest(r.inline)], 1).many(1),
mark,
]).map(result => {
if (typeof result === 'string') return result;
return M.BOLD(mergeText(result[1] as (M.MfmInline | string)[]));
});
},
boldTag: r => {
const open = P.str('<b>');
const close = P.str('</b>');
return seqOrText([
open,
P.seq([P.notMatch(close), nest(r.inline)], 1).many(1),
close,
]).map(result => {
if (typeof result === 'string') return result;
return M.BOLD(mergeText(result[1] as (M.MfmInline | string)[]));
});
},
boldUnder: r => {
const mark = P.str('__');
return P.seq([
mark,
P.alt([alphaAndNum, space]).many(1),
mark,
]).map(result => M.BOLD(mergeText(result[1] as string[])));
},
smallTag: r => {
const open = P.str('<small>');
const close = P.str('</small>');
return seqOrText([
open,
P.seq([P.notMatch(close), nest(r.inline)], 1).many(1),
close,
]).map(result => {
if (typeof result === 'string') return result;
return M.SMALL(mergeText(result[1] as (M.MfmInline | string)[]));
});
},
italicTag: r => {
const open = P.str('<i>');
const close = P.str('</i>');
return seqOrText([
open,
P.seq([P.notMatch(close), nest(r.inline)], 1).many(1),
close,
]).map(result => {
if (typeof result === 'string') return result;
return M.ITALIC(mergeText(result[1] as (M.MfmInline | string)[]));
});
},
italicAsta: r => {
const mark = P.str('*');
const parser = P.seq([
mark,
P.alt([alphaAndNum, space]).many(1),
mark,
]);
return new P.Parser((input, index, state) => {
const result = parser.handler(input, index, state);
if (!result.success) {
return P.failure();
}
// check before
const beforeStr = input.slice(0, index);
if (/[a-z0-9]$/i.test(beforeStr)) {
return P.failure();
}
return P.success(result.index, M.ITALIC(mergeText(result.value[1] as string[])));
});
},
italicUnder: r => {
const mark = P.str('_');
const parser = P.seq([
mark,
P.alt([alphaAndNum, space]).many(1),
mark,
]);
return new P.Parser((input, index, state) => {
const result = parser.handler(input, index, state);
if (!result.success) {
return P.failure();
}
// check before
const beforeStr = input.slice(0, index);
if (/[a-z0-9]$/i.test(beforeStr)) {
return P.failure();
}
return P.success(result.index, M.ITALIC(mergeText(result.value[1] as string[])));
});
},
strikeTag: r => {
const open = P.str('<s>');
const close = P.str('</s>');
return seqOrText([
open,
P.seq([P.notMatch(close), nest(r.inline)], 1).many(1),
close,
]).map(result => {
if (typeof result === 'string') return result;
return M.STRIKE(mergeText(result[1] as (M.MfmInline | string)[]));
});
},
strikeWave: r => {
const mark = P.str('~~');
return seqOrText([
mark,
P.seq([P.notMatch(P.alt([mark, newLine])), nest(r.inline)], 1).many(1),
mark,
]).map(result => {
if (typeof result === 'string') return result;
return M.STRIKE(mergeText(result[1] as (M.MfmInline | string)[]));
});
},
unicodeEmoji: r => {
const emoji = RegExp(twemojiRegex.source);
return P.regexp(emoji).map(content => M.UNI_EMOJI(content));
},
plainTag: r => {
const open = P.str('<plain>');
const close = P.str('</plain>');
return P.seq([
open,
newLine.option(),
P.seq([
P.notMatch(P.seq([newLine.option(), close])),
P.char,
], 1).many(1).text(),
newLine.option(),
close,
], 2).map(result => M.PLAIN(result));
},
fn: r => {
const fnName = new P.Parser((input, index, state) => {
const result = P.regexp(/[a-z0-9_]+/i).handler(input, index, state);
if (!result.success) {
return result;
}
if (state.fnNameList != null && !state.fnNameList.includes(result.value)) {
return P.failure();
}
return P.success(result.index, result.value);
});
const arg: P.Parser<ArgPair> = P.seq([
P.regexp(/[a-z0-9_]+/i),
P.seq([
P.str('='),
P.regexp(/[a-z0-9_.]+/i),
], 1).option(),
]).map(result => {
return {
k: result[0],
v: (result[1] != null) ? result[1] : true,
};
});
const args = P.seq([
P.str('.'),
arg.sep(P.str(','), 1),
], 1).map(pairs => {
const result: Args = { };
for (const pair of pairs) {
result[pair.k] = pair.v;
}
return result;
});
const fnClose = P.str(']');
return seqOrText([
P.str('$['),
fnName,
args.option(),
P.str(' '),
P.seq([P.notMatch(fnClose), nest(r.inline)], 1).many(1),
fnClose,
]).map(result => {
if (typeof result === 'string') return result;
const name = result[1];
const args = result[2] || {};
const content = result[4];
return M.FN(name, args, mergeText(content));
});
},
inlineCode: r => {
const mark = P.str('`');
return P.seq([
mark,
P.seq([
P.notMatch(P.alt([mark, P.str('´'), newLine])),
P.char,
], 1).many(1),
mark,
]).map(result => M.INLINE_CODE(result[1].join('')));
},
mathInline: r => {
const open = P.str('\\(');
const close = P.str('\\)');
return P.seq([
open,
P.seq([
P.notMatch(P.alt([close, newLine])),
P.char,
], 1).many(1),
close,
]).map(result => M.MATH_INLINE(result[1].join('')));
},
mention: r => {
const parser = P.seq([
notLinkLabel,
P.str('@'),
P.regexp(/[a-z0-9_-]+/i),
P.seq([
P.str('@'),
P.regexp(/[a-z0-9_.-]+/i),
], 1).option(),
]);
return new P.Parser<M.MfmMention | string>((input, index, state) => {
let result;
result = parser.handler(input, index, state);
if (!result.success) {
return P.failure();
}
// check before (not mention)
const beforeStr = input.slice(0, index);
if (/[a-z0-9]$/i.test(beforeStr)) {
return P.failure();
}
let invalidMention = false;
const resultIndex = result.index;
const username: string = result.value[2];
const hostname: string | null = result.value[3];
// remove [.-] of tail of hostname
let modifiedHost = hostname;
if (hostname != null) {
result = /[.-]+$/.exec(hostname);
if (result != null) {
modifiedHost = hostname.slice(0, (-1 * result[0].length));
if (modifiedHost.length === 0) {
// disallow invalid char only hostname
invalidMention = true;
modifiedHost = null;
}
}
}
// remove "-" of tail of username
let modifiedName = username;
result = /-+$/.exec(username);
if (result != null) {
if (modifiedHost == null) {
modifiedName = username.slice(0, (-1 * result[0].length));
} else {
// cannnot to remove tail of username if exist hostname
invalidMention = true;
}
}
// disallow "-" of head of username
if (modifiedName.length === 0 || modifiedName[0] === '-') {
invalidMention = true;
}
// disallow [.-] of head of hostname
if (modifiedHost != null && /^[.-]/.test(modifiedHost)) {
invalidMention = true;
}
// generate a text if mention is invalid
if (invalidMention) {
return P.success(resultIndex, input.slice(index, resultIndex));
}
const acct = modifiedHost != null ? `@${modifiedName}@${modifiedHost}` : `@${modifiedName}`;
return P.success(index + acct.length, M.MENTION(modifiedName, modifiedHost, acct));
});
},
hashtag: r => {
const mark = P.str('#');
const hashTagChar = P.seq([
P.notMatch(P.alt([P.regexp(/[ \u3000\t.,!?'"#:/[\]【】()「」（）<>]/), space, newLine])),
P.char,
], 1);
const innerItem: P.Parser<any> = P.lazy(() => P.alt([
P.seq([
P.str('('), nest(innerItem, hashTagChar).many(0), P.str(')'),
]),
P.seq([
P.str('['), nest(innerItem, hashTagChar).many(0), P.str(']'),
]),
P.seq([
P.str('「'), nest(innerItem, hashTagChar).many(0), P.str('」'),
]),
P.seq([
P.str('（'), nest(innerItem, hashTagChar).many(0), P.str('）'),
]),
hashTagChar,
]));
const parser = P.seq([
notLinkLabel,
mark,
innerItem.many(1).text(),
], 2);
return new P.Parser((input, index, state) => {
const result = parser.handler(input, index, state);
if (!result.success) {
return P.failure();
}
// check before
const beforeStr = input.slice(0, index);
if (/[a-z0-9]$/i.test(beforeStr)) {
return P.failure();
}
const resultIndex = result.index;
const resultValue = result.value;
// disallow number only
if (/^[0-9]+$/.test(resultValue)) {
return P.failure();
}
return P.success(resultIndex, M.HASHTAG(resultValue));
});
},
emojiCode: r => {
const mark = P.str(':');
return P.seq([
mark,
P.regexp(/[a-z0-9_+-]+/i),
mark,
], 1).map(name => M.EMOJI_CODE(name as string));
},
link: r => {
const labelInline = new P.Parser((input, index, state) => {
state.linkLabel = true;
const result = r.inline.handler(input, index, state);
state.linkLabel = false;
return result;
});
const closeLabel = P.str(']');
return P.seq([
notLinkLabel,
P.alt([P.str('?['), P.str('[')]),
P.seq([
P.notMatch(P.alt([closeLabel, newLine])),
nest(labelInline),
], 1).many(1),
closeLabel,
P.str('('),
P.alt([r.urlAlt, r.url]),
P.str(')'),
]).map(result => {
const silent = (result[1] === '?[');
const label = result[2];
const url: M.MfmUrl = result[5];
return M.LINK(silent, url.props.url, mergeText(label));
});
},
url: r => {
const urlChar = P.regexp(/[.,a-z0-9_/:%#@$&?!~=+-]/i);
const innerItem: P.Parser<any> = P.lazy(() => P.alt([
P.seq([
P.str('('), nest(innerItem, urlChar).many(0), P.str(')'),
]),
P.seq([
P.str('['), nest(innerItem, urlChar).many(0), P.str(']'),
]),
urlChar,
]));
const parser = P.seq([
notLinkLabel,
P.regexp(/https?:\/\//),
innerItem.many(1).text(),
]);
return new P.Parser<M.MfmUrl | string>((input, index, state) => {
let result;
result = parser.handler(input, index, state);
if (!result.success) {
return P.failure();
}
const resultIndex = result.index;
let modifiedIndex = resultIndex;
const schema: string = result.value[1];
let content: string = result.value[2];
// remove the ".," at the right end
result = /[.,]+$/.exec(content);
if (result != null) {
modifiedIndex -= result[0].length;
content = content.slice(0, (-1 * result[0].length));
if (content.length === 0) {
return P.success(resultIndex, input.slice(index, resultIndex));
}
}
return P.success(modifiedIndex, M.N_URL(schema + content, false));
});
},
urlAlt: r => {
const open = P.str('<');
const close = P.str('>');
const parser = P.seq([
notLinkLabel,
open,
P.regexp(/https?:\/\//),
P.seq([P.notMatch(P.alt([close, space])), P.char], 1).many(1),
close,
]).text();
return new P.Parser((input, index, state) => {
const result = parser.handler(input, index, state);
if (!result.success) {
return P.failure();
}
const text = result.value.slice(1, (result.value.length - 1));
return P.success(result.index, M.N_URL(text, true));
});
},
search: r => {
const button = P.alt([
P.regexp(/\[(検索|search)\]/i),
P.regexp(/(検索|search)/i),
]);
return P.seq([
newLine.option(),
P.lineBegin,
P.seq([
P.notMatch(P.alt([
newLine,
P.seq([space, button, P.lineEnd]),
])),
P.char,
], 1).many(1),
space,
button,
P.lineEnd,
newLine.option(),
]).map(result => {
const query = result[2].join('');
return M.SEARCH(query, `${query}${result[3]}${result[4]}`);
});
},
text: r => P.char,
});

48
src/internal/util.ts
View file

@ -1,7 +1,7 @@
import { isMfmBlock, MfmNode, TEXT } from '../node'; import { isMfmBlock, MfmInline, MfmNode, MfmText, TEXT } from '../node';
export function mergeText(nodes: (MfmNode | string)[]): MfmNode[] { export function mergeText<T extends MfmNode>(nodes: ((T extends MfmInline ? MfmInline : MfmNode) | string)[]): (T | MfmText)[] {
const dest: MfmNode[] = []; const dest: (T | MfmText)[] = [];
const storedChars: string[] = []; const storedChars: string[] = [];
/** /**
@ -14,11 +14,15 @@ export function mergeText(nodes: (MfmNode | string)[]): MfmNode[] {
} }
} }
for (const node of nodes) { const flatten = nodes.flat(1) as (string | T)[];
for (const node of flatten) {
if (typeof node === 'string') { if (typeof node === 'string') {
// Store the char. // Store the char.
storedChars.push(node); storedChars.push(node);
} }
else if (!Array.isArray(node) && node.type === 'text') {
storedChars.push((node as MfmText).props.text);
}
else { else {
generateText(); generateText();
dest.push(node); dest.push(node);
@ -163,39 +167,3 @@ export function inspectOne(node: MfmNode, action: (node: MfmNode) => void) {
} }
} }
} }
//
// dynamic consuming
//
/*
1. If you want to consume 3 chars, call the setConsumeCount.
```
setConsumeCount(3);
```
2. And the rule to consume the input is as below:
```
rule = (&{ return consumeDynamically(); } .)+
```
*/
let consumeCount = 0;
/**
* set the length of dynamic consuming.
*/
export function setConsumeCount(count: number) {
consumeCount = count;
}
/**
* consume the input and returns matching result.
*/
export function consumeDynamically() {
const matched = (consumeCount > 0);
if (matched) {
consumeCount--;
}
return matched;
}

134
test/parser.ts
View file

@ -143,6 +143,24 @@ hoge`;
]; ];
assert.deepStrictEqual(mfm.parse(input), output); assert.deepStrictEqual(mfm.parse(input), output);
}); });
it('2つの引用行の間に空行がある場合は2つの引用ブロックが生成される', () => {
const input = `
> foo
> bar
hoge`;
const output = [
QUOTE([
TEXT('foo')
]),
QUOTE([
TEXT('bar')
]),
TEXT('hoge'),
];
assert.deepStrictEqual(mfm.parse(input), output);
});
}); });
describe('search', () => { describe('search', () => {
@ -684,6 +702,60 @@ hoge`;
const output = [TEXT('あいう'), MENTION('abc', null, '@abc')]; const output = [TEXT('あいう'), MENTION('abc', null, '@abc')];
assert.deepStrictEqual(mfm.parse(input), output); assert.deepStrictEqual(mfm.parse(input), output);
}); });
it('invalid char only username', () => {
const input = '@-';
const output = [TEXT('@-')];
assert.deepStrictEqual(mfm.parse(input), output);
});
it('invalid char only hostname', () => {
const input = '@abc@.';
const output = [TEXT('@abc@.')];
assert.deepStrictEqual(mfm.parse(input), output);
});
it('allow "-" in username', () => {
const input = '@abc-d';
const output = [MENTION('abc-d', null, '@abc-d')];
assert.deepStrictEqual(mfm.parse(input), output);
});
it('disallow "-" in head of username', () => {
const input = '@-abc';
const output = [TEXT('@-abc')];
assert.deepStrictEqual(mfm.parse(input), output);
});
it('disallow "-" in tail of username', () => {
const input = '@abc-';
const output = [MENTION('abc', null, '@abc'), TEXT('-')];
assert.deepStrictEqual(mfm.parse(input), output);
});
it('disallow "." in head of hostname', () => {
const input = '@abc@.aaa';
const output = [TEXT('@abc@.aaa')];
assert.deepStrictEqual(mfm.parse(input), output);
});
it('disallow "." in tail of hostname', () => {
const input = '@abc@aaa.';
const output = [MENTION('abc', 'aaa', '@abc@aaa'), TEXT('.')];
assert.deepStrictEqual(mfm.parse(input), output);
});
it('disallow "-" in head of hostname', () => {
const input = '@abc@-aaa';
const output = [TEXT('@abc@-aaa')];
assert.deepStrictEqual(mfm.parse(input), output);
});
it('disallow "-" in tail of hostname', () => {
const input = '@abc@aaa-';
const output = [MENTION('abc', 'aaa', '@abc@aaa'), TEXT('-')];
assert.deepStrictEqual(mfm.parse(input), output);
});
}); });
describe('hashtag', () => { describe('hashtag', () => {
@ -847,6 +919,14 @@ hoge`;
assert.deepStrictEqual(mfm.parse(input), output); assert.deepStrictEqual(mfm.parse(input), output);
}); });
it('disallow period only', () => {
const input = 'https://.';
const output = [
TEXT('https://.')
];
assert.deepStrictEqual(mfm.parse(input), output);
});
it('ignore trailing periods', () => { it('ignore trailing periods', () => {
const input = 'https://misskey.io/@ai...'; const input = 'https://misskey.io/@ai...';
const output = [ const output = [
@ -1107,6 +1187,14 @@ hoge`;
assert.deepStrictEqual(mfm.parse(input), output); assert.deepStrictEqual(mfm.parse(input), output);
}); });
it('invalid fn name', () => {
const input = '$[関数 text]';
const output = [
TEXT('$[関数 text]')
];
assert.deepStrictEqual(mfm.parse(input), output);
});
it('nest', () => { it('nest', () => {
const input = '$[spin.speed=1.1s $[shake a]]'; const input = '$[spin.speed=1.1s $[shake a]]';
const output = [ const output = [
@ -1179,11 +1267,7 @@ hoge`;
const output = [ const output = [
QUOTE([ QUOTE([
QUOTE([ QUOTE([
TEXT('*'), TEXT('**abc**'),
ITALIC([
TEXT('abc'),
]),
TEXT('*'),
]), ]),
]), ]),
]; ];
@ -1196,11 +1280,7 @@ hoge`;
const output = [ const output = [
BOLD([ BOLD([
BOLD([ BOLD([
TEXT('**'), TEXT('***abc***'),
ITALIC([
TEXT('abc'),
]),
TEXT('**'),
]), ]),
]), ]),
]; ];
@ -1213,11 +1293,7 @@ hoge`;
const output = [ const output = [
ITALIC([ ITALIC([
ITALIC([ ITALIC([
TEXT('*'), TEXT('**abc**'),
ITALIC([
TEXT('abc'),
]),
TEXT('*'),
]), ]),
]), ]),
]; ];
@ -1289,13 +1365,19 @@ hoge`;
describe('hashtag', () => { describe('hashtag', () => {
it('basic', () => { it('basic', () => {
const input = '<b><b>#abc(xyz)</b></b>'; let input, output;
const output = [ input = '<b>#abc(xyz)</b>';
output = [
BOLD([ BOLD([
HASHTAG('abc(xyz)'),
]),
];
assert.deepStrictEqual(mfm.parse(input, { nestLimit: 2 }), output);
input = '<b>#abc(x(y)z)</b>';
output = [
BOLD([ BOLD([
HASHTAG('abc'), HASHTAG('abc'),
TEXT('(xyz)'), TEXT('(x(y)z)'),
]),
]), ]),
]; ];
assert.deepStrictEqual(mfm.parse(input, { nestLimit: 2 }), output); assert.deepStrictEqual(mfm.parse(input, { nestLimit: 2 }), output);
@ -1343,13 +1425,19 @@ hoge`;
}); });
it('url', () => { it('url', () => {
const input = '<b><b>https://example.com/abc(xyz)</b></b>'; let input, output;
const output = [ input = '<b>https://example.com/abc(xyz)</b>';
output = [
BOLD([ BOLD([
N_URL('https://example.com/abc(xyz)'),
]),
];
assert.deepStrictEqual(mfm.parse(input, { nestLimit: 2 }), output);
input = '<b>https://example.com/abc(x(y)z)</b>';
output = [
BOLD([ BOLD([
N_URL('https://example.com/abc'), N_URL('https://example.com/abc'),
TEXT('(xyz)'), TEXT('(x(y)z)'),
]),
]), ]),
]; ];
assert.deepStrictEqual(mfm.parse(input, { nestLimit: 2 }), output); assert.deepStrictEqual(mfm.parse(input, { nestLimit: 2 }), output);

4
tsconfig.json
View file

@ -12,6 +12,10 @@
"noImplicitReturns": true, "noImplicitReturns": true,
"esModuleInterop": true, "esModuleInterop": true,
}, },
"typeRoots": [
"node_modules/@types",
"src/@types",
],
"include": [ "include": [
"src/**/*", "src/**/*",
], ],